A multi-modal recommendation method based on multi-source neighborhood conditional diffusion

Abstract

The application discloses a kind of multi-modal recommendation method based on multi-source neighborhood condition diffusion, comprising: generating noisy representation input condition diffusion representation refinement network, obtain prediction representation, calculate diffusion loss according to the difference between target item latent representation and prediction representation;After performing conditional iterative update, obtain item node initial representation;It is input with user embedding vector to interact graph to carry out neighborhood information propagation, calculate ranking loss according to the final user representation and final item representation obtained;According to diffusion loss and ranking loss, the parameter of network is updated, to obtain updated network;Recommendation score is obtained by updated condition diffusion representation refinement network, and further obtain item recommendation result.The application is helpful to alleviate the representation convergence problem, can improve the discrimination ability and the accuracy of recommendation result, can reduce the number of iterations in the representation enhancement process, reduce the computing cost and inference delay, improve the applicability in large-scale recommendation scene.

Description

Technical Field

[0001] This invention belongs to the field of data mining and intelligent recommendation technology, specifically relating to a multimodal recommendation method based on multi-source neighborhood conditional diffusion. Background Technology

[0002] In recent years, the scale of information such as goods, videos, news, and books on internet platforms has continued to grow. How to identify content that users are interested in from a large number of candidate items has become an important problem for intelligent recommendation technology. Recommendation systems typically learn user preference representations and item feature representations based on historical user-item interaction data, and predict the degree of user preference for candidate items.

[0003] To alleviate the problems of sparse interactive data and cold start for new items, multimodal recommendation methods incorporate content features such as text and images of items. Some methods construct item-item semantic neighborhood graphs based on the similarity of multimodal features and utilize graph neural networks to propagate information, thereby enhancing the item representation capabilities.

[0004] However, existing multimodal recommendation methods based on item neighborhood graphs still have limitations. Semantic neighborhood graphs built solely based on content similarity are easily affected by factors such as text redundancy and image background noise, making it difficult to characterize the co-occurrence relationships of items in interaction data; behavioral neighborhood graphs built solely based on interaction data are difficult to represent the potential associations between non-interacting items. Furthermore, fixed weighting rules or simple concatenation methods struggle to adjust the contributions of different neighborhoods according to the characteristics of the target item. In addition, traditional graph aggregation methods typically use fixed propagation rules to aggregate neighbor node information, making it difficult to adaptively adjust neighbor contributions based on differences in the target item, easily leading to convergence in item representations and weakening the model's ability to distinguish fine-grained semantic differences. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a multimodal recommendation method based on multi-source neighborhood conditional diffusion.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a multimodal recommendation method based on multi-source neighborhood conditional diffusion, the method comprising: A noise-added representation for the corresponding diffusion time step is generated based on a preset noise scheduling strategy; The network is refined by combining the noisy representation, the multi-source neighborhood conditional representation, and the temporal embedding input conditional diffusion representation corresponding to the diffusion time step to obtain the predicted representation of the target item. The diffusion loss is then calculated based on the difference between the latent representation of the target item and the predicted representation. The latent representation of the target item is obtained by mapping the multimodal features of the target item. Based on the conditional diffusion representation refinement network and the multi-source neighborhood conditional representation of the target item, the latent representation of the target item is updated conditionally at multiple time steps to obtain the enhanced item representation; The enhanced item representation is added to the latent representation of the target item by residual addition to obtain the initial representation of the item node; The initial representation of the item node is input into the interaction graph with the user embedding vector to propagate neighborhood information, and the ranking loss is calculated based on the final user representation and the final item representation. The parameters of the conditional diffusion representation refinement network are updated based on the diffusion loss and the ranking loss to obtain the updated conditional diffusion representation refinement network. The updated conditional diffusion representation refines the network to obtain the final candidate item representation and the corresponding end-user representation, and performs an inner product to obtain a recommendation score. The item recommendation results are then obtained by sorting the items according to the recommendation scores.

[0007] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: In the above technical solution, this invention uses multi-source neighborhood information as a condition to guide the conditional diffusion representation refinement network to perform multi-time-step iterative denoising and representation refinement on the latent representation of items. Compared with traditional fixed graph aggregation rules, this invention helps alleviate the representation convergence problem in the neighborhood aggregation process by progressively modifying and enhancing the item representation, thereby improving the model's ability to distinguish fine-grained semantic differences. This invention fuses the enhanced item representation with the latent representation of the target item through residual connections and propagates neighborhood information on the user-item interaction graph, thus incorporating user interaction preferences while preserving the multimodal features of the item itself. The joint optimization of ranking loss and diffusion loss keeps the representation learning process consistent with the recommendation goal, which is beneficial to improving the accuracy of the recommendation results. Furthermore, this invention can adopt a step-by-step conditional iterative update, which can reduce the number of iterations in the representation enhancement process, reduce computational overhead and inference latency, and improve its applicability in large-scale recommendation scenarios.

[0008] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0009] Figure 1 This is a flowchart of a multimodal recommendation method based on multi-source neighborhood conditional diffusion provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the forward and backward processes of a diffusion model provided in an embodiment of the present invention. Detailed Implementation

[0010] To facilitate understanding of the present invention, a brief description of the prior art and the inventive concept of the present invention will be provided first.

[0011] Diffusion models can progressively refine data representations through a multi-step denoising process. Introducing the concept of conditional diffusion into multimodal recommendation scenarios allows multi-source neighborhood information to be used as conditions to guide iterative denoising and refinement of latent item representations, thereby correcting and enhancing item representations.

[0012] Therefore, how to integrate semantic neighborhood relationships formed by content similarity and behavioral neighborhood relationships formed by user interaction behavior, and how to adaptively and iteratively refine item representations using conditional diffusion processes, are problems that need to be further solved in current multimodal recommendation technologies. This invention proposes a multimodal recommendation method based on multi-source neighborhood conditional diffusion to address this technical problem.

[0013] Figure 1 This is a flowchart of a multimodal recommendation method based on multi-source neighborhood conditional diffusion provided by an embodiment of the present invention, such as... Figure 1 As shown, the method may include the following steps: S101. Generate a noise representation of the corresponding diffusion time step according to the preset noise scheduling strategy.

[0014] Figure 2 This is a schematic diagram of the forward and backward processes of a diffusion model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, in the forward process of the diffusion model, noise is gradually injected into the latent representation of the item; in the backward process of the diffusion model, the multi-source neighborhood conditional representation is used as guiding information, and the original item features are iteratively recovered from the noisy latent representation of the item through a parameterized neural network. Furthermore, the loss function is calculated based on the mean square error, and the loss function is backpropagated to train the diffusion model.

[0015] For modal The target item to represent the potential of items The target representation for denoising is determined as follows: ; From the preset diffusion time step set Randomly select time steps Gaussian noise is added to the denoised target representation based on the cumulative noise preservation coefficient corresponding to the time step, resulting in a noisy representation. : ; in, , This represents the total number of diffusion time steps. Indicates time step The corresponding cumulative noise retention factor.

[0016] Optionally, prior to S101, the method may further include: Obtain multimodal features of the item set and user-item interaction data, and construct an interaction graph based on the interaction data; A semantic neighborhood graph is constructed based on the similarity of item content features between different items in the same modality obtained from multimodal features, and a behavioral neighborhood graph is constructed based on the behavioral association strength between different items obtained from interaction data. Map the multimodal features of items in the item set to latent representations of the items; Extract the semantic neighborhood relation vector and behavioral neighborhood relation vector corresponding to the target item in the item set based on the semantic neighborhood graph and behavioral neighborhood graph; The semantic neighborhood relation vector and the behavioral neighborhood relation vector are weighted and fused to obtain the fused neighborhood relation vector. The multi-source neighborhood conditional representation of the target item is obtained by weighted aggregation based on the item's latent representation and the fused neighborhood relation vector.

[0017] Optionally, a semantic neighborhood graph is constructed based on the similarity of item content features among different items in the same modality obtained from multimodal features, and a behavioral neighborhood graph is constructed based on the behavioral association strength among different items obtained from interaction data, including: Extract the similarity of item content features between the target item and any other item in the item set under the same modality based on multimodal features; For the target item, after sorting the similarity of the item content features in descending order, a preset number of item connections are retained and normalized to obtain a semantic neighborhood graph. The behavioral association strength between the target item and any other item in the item set is obtained based on the first set of interacting users corresponding to the target item and the second set of interacting users corresponding to any other item in the item set. For the target item, after sorting the behavior association strength in descending order, a preset number of item connections greater than the preset filtering threshold are retained and normalized to obtain the behavior neighborhood graph.

[0018] It is understandable that obtaining the user set Item collection User-Item Interaction Matrix Among them, the user-item interaction matrix Its elements Indicates user With the target item There is an interactive relationship. Indicates user With the target item There is no interaction relationship; based on the user-item interaction matrix. Constructing a user-item interaction graph , where the set of nodes Includes user nodes and item nodes, and an edge set. This includes the connection relationship between user nodes and interacted item nodes; obtaining the target item. In modality Multimodal features , among which, modality Includes text-based and visual modalities; initializes the user. In modality User embedding vectors .

[0019] For any mode According to the item With the target item In modality Pre-extracted multimodal features are used to calculate the items. With the target item Content feature similarity between them: ; in, and Representing items and target items In modality Pre-extracted multimodal features Represents items With the target item In modality The similarity of content features.

[0020] For each target item Retain the content with the highest similarity to the first preset number of elements. The modal is obtained by deleting the connections between the target item and itself, and normalizing the remaining item connections. The adjacency matrix of the corresponding semantic neighborhood graph ,in, This represents the number of semantic neighbors retained for each target item; Specifically, for a semantic neighborhood graph, the similarity of content features after neighbor filtering and self-join removal is normalized to obtain the adjacency matrix of the semantic neighborhood graph. : ; in, This indicates that items have been filtered and deleted after self-connection by neighbors. With the target item In modality Content feature similarity Represents items For target items Semantic neighborhood contribution weight; Based on user-item interaction data, determine the relationship with the item. The set of users who interacted and the target item The set of users who interacted And calculate the items based on the size of the intersection between user sets. With the target item Strength of behavioral association between them: ; in, Represents items With the target item The strength of the behavioral association between them; For each target item The retention behavior is ranked first in terms of correlation strength. The position is greater than the preset filtering threshold. The item connection relationships are determined by deleting the connection relationships between the target item and itself, and then normalizing the remaining item connections to obtain the adjacency matrix of the behavior neighborhood graph. ,in, This represents the preset filtering threshold used to filter out low-intensity behavioral connections.

[0021] Specifically, for the behavior neighborhood graph, the behavior association strength after neighbor filtering, threshold filtering, and self-join removal is normalized to obtain the adjacency matrix of the behavior neighborhood graph. : ; in, This indicates items that have undergone neighbor filtering, threshold filtering, and removal of self-joins. With the target item The strength of behavioral association between Represents items For target items The neighborhood contribution weight of behavior This indicates the number of items in the collection.

[0022] The multimodal features of an item are mapped to its latent representation. For the target item, the corresponding trainable fusion parameters are used to perform weighted fusion of the neighborhood relationships corresponding to the target item in the semantic neighborhood graph and the behavioral neighborhood graph. The latent representations of neighboring items are then weighted and aggregated based on the fused neighborhood relationships to obtain a multi-source neighborhood conditional representation. For modal , target item Multimodal features Input and Modality The corresponding feature mapping network is used, and the mapping results are normalized to obtain the latent representation of the target item. : ; in, Representation and Modality The corresponding feature mapping network, Indicates normalization processing; target items are extracted from the semantic neighborhood map and the behavioral neighborhood map, respectively. The corresponding semantic neighborhood relation vector and behavioral neighborhood relation vector ; Obtain the target item Corresponding trainable fusion parameters Based on trainable fusion parameters semantic neighborhood relation vector and behavioral neighborhood relation vector Perform weighted fusion to obtain the target item. In modality The fusion neighborhood relation vector : ; in, Indicates the target item The fusion parameters of the corresponding semantic neighborhood relationship and behavioral neighborhood relationship.

[0023] modality The latent representation matrix consists of the latent representations of each item. And based on the fusion neighborhood relation vector The target item is obtained by weighted aggregation of the potential representations of neighboring items. In modality Multi-source neighborhood condition representation : ; Among them, multi-source neighborhood condition representation This network guides the Conditional Diffusion Representation Refinement Network (CDRN) to iteratively update the latent representation of the target item. The CDRN is a conditional denoising prediction network in the form of a multilayer perceptron, comprising a temporal embedding layer, an input concatenation layer, and a denoising prediction layer composed of multiple fully connected layers. The temporal embedding layer converts the diffusion time steps into temporal step embeddings. The input concatenation layer concatenates the noisy item representation, the temporal step embeddings, and the multi-source neighborhood conditional representation. The denoising prediction layer performs feature transformation on the concatenated input; the first half of the fully connected layer extracts and compresses fused features, while the second half performs dimensionality restoration and outputs a predicted representation consistent with the latent representation of the item.

[0024] S102. The network is refined by adding noise representation, multi-source neighborhood conditional representation and time embedding input conditional diffusion representation corresponding to diffusion time step to obtain the prediction representation of the target item, and the diffusion loss is calculated based on the difference between the latent representation and the prediction representation of the target item; wherein, the latent representation of the target item is obtained by mapping the multimodal features of the target item.

[0025] Adding noise Multi-source neighborhood condition representation and time step The corresponding temporal embedding input conditional diffusion representation refinement network yields a predicted representation of the denoising target: ; The modality is calculated based on the difference between the predicted representation and the denoised target representation. Corresponding diffusion loss: ; in, Representing modes The diffusion loss below, Indicates the quantity of the target item. Indicates the first The potential representation of an item corresponding to a target item. Indicates the first The target item is in the first Noise representation at each time step Indicates the first The target item in the modality The multi-source neighborhood condition representation under the following conditions Indicates the first The loss weights corresponding to each time step Indicates the first The prediction representation corresponding to each time step. It represents the 2-norm.

[0026] S103. Based on the conditional diffusion representation, refine the network and the multi-source neighborhood conditional representation of the target item, and perform conditional iterative updates of the latent representation of the target item at multiple time steps to obtain the enhanced item representation.

[0027] Optionally, S103 may include: Based on the conditional diffusion representation refinement network, the conditional prediction result and the unconditional prediction result are obtained by taking the multi-source neighborhood conditional representation and the empty conditional representation as inputs, respectively; where the empty conditional representation is a zero vector with the same dimension as the multi-source neighborhood conditional representation. Based on the conditional guidance intensity parameter, the conditional prediction results and the unconditional prediction results are linearly combined to obtain the target representation prediction results; Based on the prediction results of the target representation, perform conditional iterative updates at multiple time steps to obtain the enhanced item representation.

[0028] Understandably, the conditional diffusion representation refinement network employs a joint training approach that combines conditional information preservation and conditional information removal. This joint training approach includes: During training, according to preset probabilities Representing multi-source neighborhood conditions Replace with an empty conditional representation, where the empty conditional representation is an all-zero vector with the same dimension as the multi-source neighborhood conditional representation; Conditional prediction results are generated based on multi-source neighborhood conditional representation, and unconditional prediction results are generated based on empty conditional representation. Guide strength parameters based on preset conditions The conditional and unconditional prediction results are linearly combined to obtain the target representation prediction result used for conditional iterative update: ; in, This indicates the probability of removing preset conditional information. The multi-source neighborhood condition represents the guiding strength parameter for the condition iterative update process.

[0029] Optionally, conditional iterative updates also include skip updates, the skip update process including: From containing Select from the complete time step sequence of each time step that contains The target time step sequence at each time point; The target item in modality The potential representation of the items is set to the representation in the target time step sequence to refine the initial state; Based on the refined initial state representation, a deterministic conditional step-by-step update is performed according to the target time step sequence; Will be completed The output state obtained after the deterministic conditional jump update is determined as the enhanced item representation.

[0030] target item In modality The following items represent potential meanings As a representation of the initial state for refinement: ; Skip-step updates can be performed using the following methods. Conditional iterative updates across multiple time steps also include an accelerated processing method based on deterministic skip-step updates. These accelerated processing methods include: From containing Select from the complete time step sequence of each time step that contains The target time step sequence at each time point: ; in, The time points in the target time step sequence are selected at equal intervals; target item In modality The underlying representation of the items is set to refine the initial state by representing the target time step sequence: ; for Perform deterministic conditional step-by-step updates according to the target time step sequence: ; in, Represents the first step in the target time step sequence At a certain point in time, Indicates a point in time The corresponding cumulative noise retention factor; Will be completed The output state obtained after a deterministic conditional jump update is determined as the enhanced item representation, in order to reduce the number of conditional iteration updates required to generate the enhanced item representation.

[0031] In one implementation, conditional iterative updates can also be performed using the following classic method for time steps. The current state is iteratively updated based on the prediction results of the target representation, as shown below: ; in, Indicates the first The output state obtained after a deterministic conditional jump step update at each time step. Indicates the first The noise preservation factor corresponding to each time step. Indicates from the first time step to the second time step. The cumulative product of the noise preservation coefficients at each time step Indicates the first The target corresponding to each time step represents the prediction result. Indicates the first The target item in the modality The multi-source neighborhood condition representation.

[0032] The output state obtained after iterative updates of conditions at multiple time steps is determined as the target item. In modality The following enhancement items are indicated: .

[0033] S104. Add the residuals of the enhanced item representation and the target item potential representation to obtain the initial representation of the item node.

[0034] Understandably, regarding modality , will users Corresponding user embedding vector As the initial representation of user nodes and the items Corresponding enhancement item representation With the potential representation of items By summing the residuals, we obtain the initial representation of the item nodes. : .

[0035] S105. Propagate neighborhood information between the initial representation of the item node and the user embedding vector input interaction graph, and calculate the ranking loss based on the obtained final user representation and final item representation.

[0036] Perform at least one level of neighborhood information propagation on the user-item interaction graph, for the first... Layer propagation, user node representation and item node representation They are respectively: ; in, Indicates to users A collection of items that have interactive relationships. Indicates with items A set of users with interactive relationships; The user node representations and item node representations obtained from each propagation layer are aggregated separately, and the aggregation results from different modalities are concatenated to obtain the final user representation. and final item representation : ; in, Indicates the number of layers of information propagation in the neighborhood. Represents a set of modes. This indicates a vector concatenation operation.

[0037] Construct training triples based on user-item interaction data Among them, users With the target item There is an interactive relationship, users With items There is no interaction relationship; According to end users Target items Corresponding final item representation and items Corresponding final item representation The ranking loss is expressed as follows: ; in, Represents the sorting loss. Indicates training triplet The set that is formed This represents the Sigmoid function. Indicates transpose. Indicates to users Target items with interactive relationships The final item represents, Indicates to users Items that do not have an interaction relationship The final item represents, This represents the end-user's perspective.

[0038] The ranking loss is combined with the diffusion loss corresponding to each modality to obtain the total training loss of the model: ; in, This represents the loss weight parameter corresponding to the diffusion loss.

[0039] S106. Update the parameters of the conditional diffusion representation refinement network according to the diffusion loss and ranking loss to obtain the updated conditional diffusion representation refinement network.

[0040] S107. The network is refined by the updated conditional diffusion representation to obtain the final candidate item representation and the corresponding final user representation, and the inner product is performed to obtain the recommendation score. The item recommendation result is obtained by sorting the items according to the recommendation score.

[0041] Understandably, this recommended score The calculation formula is expressed as follows:

[0042] Candidate items were ranked according to their recommended scores. Sort in descending order and place the recommended scores first. Candidate items for position Determined to be user-oriented The item recommendation results, among which, This indicates the number of recommended items to be output.

[0043] This embodiment conducts simulation experiments on the aforementioned multimodal recommendation method based on multi-source neighborhood conditional diffusion, and compares it with existing multimodal recommendation methods to further illustrate the effectiveness of the present invention.

[0044] 1. Simulation Experiment Conditions: Operating System: Ubuntu 20.04, Python 3.12.9; Experimental Platform: PyTorch 2.8.0; Processor: Intel Core i9-14900KF; Graphics Card: NVIDIA RTX 5070 Ti; Memory: 16GB; 2. Simulation Experiment Content: Simulation Experiment 1: Comparison of Recall and Normalized Decline Cumulative Gain for Recommendation Tasks It should be noted that all the following experiments were conducted in the same experimental environment. The dataset used is a classic dataset for multimodal recommendation methods. Both the benchmark method and the multimodal recommendation method with multi-source neighborhood conditional diffusion proposed in this invention belong to graph learning-based multimodal recommendation methods.

[0045] Table 1

[0046] Table 1 compares the recall and normalized cumulative gain (N@K) of the baseline method and the method proposed in this invention. As shown in Table 1, the experimental results compare the performance of the baseline method and the method proposed in this invention in terms of recall (R@K) and normalized cumulative gain (N@K). The experimental results are as follows: The method proposed in this invention outperforms the existing best-in-class methods on both the Baby and Electronics datasets. Specifically, on the Baby dataset, the method improves the R@10 and N@10 metrics by 7.07% and 7.95%, respectively; on the Electronics dataset, it improves the R@10 and R@20 metrics by 7.05% and 6.41%, respectively. These results demonstrate that the method proposed in this invention can stably improve recommendation performance on datasets of different sizes, not only increasing the proportion of real-world interactive items hit in the recommended list but also improving the ranking quality of the recommendation results. This result verifies the effectiveness of this invention in enhancing item representation using multi-source neighborhood conditional diffusion.

[0047] Simulation Experiment 2: Comparison Results of Full Diffusion Update and Skip-Step Update Methods To verify the efficiency improvement effect of the deterministic skip-step update method in this invention, this embodiment compares the full diffusion update method with the skip-step update method. The full diffusion update method performs all reverse conditional iterative updates according to a preset diffusion time step sequence; the skip-step update method selects a smaller number of target time points from the full time step sequence and performs deterministic conditional skip-step updates according to the target time points, thereby reducing the number of diffusion updates required during representation enhancement.

[0048] Table 2

[0049] Table 2 compares the full diffusion update and skip-diffusion update methods in this invention. As shown in Table 2, the experimental results compare the performance of the full update method and the skip-diffusion update method in terms of recall R@K and normalized cumulative gain N@K. Compared to the full diffusion update method, the skip-diffusion update method significantly reduces training time while maintaining relatively stable recommendation performance. On the Baby dataset, the training time per round of the skip-diffusion update method decreased from 2.67s to 2.21s, a reduction of 17.2%; on the Electronics dataset, the training time per round decreased from 131.19s to 71.66s, a reduction of 45.4%. From the perspective of recommendation performance, the skip-step update method shows a slight decrease in R@20 and N@20 on the Baby dataset, from 0.1053 and 0.0471 to 0.1034 and 0.0459, respectively. On the Electronics dataset, R@20 changes from 0.0697 to 0.0692, while N@20 remains at 0.0320. These results indicate that although the skip-step update method reduces the number of diffusion reverse updates, it does not cause a significant loss in recommendation performance. These experimental results demonstrate that the deterministic skip-step update method in this invention can effectively reduce the computational overhead of the conditional diffusion representation refinement process and exhibits a more significant acceleration effect on larger datasets. Therefore, the skip-step update method can improve the applicability of the method in large-scale multimodal recommendation scenarios.

[0050] Simulation Experiment 3: Ablation Experiment of Main Modules of Multimodal Recommendation Method Based on Multi-Source Neighborhood Conditional Diffusion To verify the contribution of each major technical module in this invention to the recommended performance, this embodiment conducts ablation experiments on the major modules under the same experimental environment and compares the changes in recommended performance of different variant methods.

[0051] Table 3

[0052] Table 3 shows the ablation experiment data based on the present invention. As shown in Table 3, the experimental results provide a performance comparison of the proposed method and its two variants in terms of recall R@K and normalized cumulative gain N@K. The removal of the diffusion representation refinement module means that the latent representation of the item is directly input into the user-item interaction graph for propagation. The removal of the behavior neighborhood graph means that the behavior neighborhood graph constructed based on the number of users with common interaction is removed, and only the semantic neighborhood graph constructed based on the similarity of content features is retained. The experimental results are as follows: (1) The removal of the diffusion representation refinement module leads to a decrease in performance, indicating that the conditional diffusion representation refinement module can effectively enhance the latent representation of the item and improve the accuracy of the recommendation results. (2) After removing the behavior neighborhood graph, the model performance also decreases, indicating that the behavior neighborhood graph constructed based on the number of users with common interaction can provide complementary information to the semantic neighborhood graph, which helps to alleviate the problem of insufficient expression of single semantic neighborhood relationships.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a semantic neighborhood graph based on content similarity and a behavioral neighborhood graph based on the number of users with common interactions, and adaptively weights different neighborhood relationships using trainable fusion parameters. This method can simultaneously utilize item content association and user behavior co-occurrence signals, alleviating the problem of insufficient information from a single neighborhood source, and improving the insufficient fusion of neighborhood information under fixed weighting or simple splicing methods.

[0054] 2. This invention uses multi-source neighborhood information as a condition to guide the conditional diffusion representation refinement network to perform multi-time-step iterative denoising and representation refinement on the latent representations of items. Compared with traditional fixed graph aggregation rules, this approach helps alleviate the representation convergence problem in the neighborhood aggregation process by progressively refining and improving the model's ability to distinguish fine-grained semantic differences.

[0055] 3. This invention fuses the enhanced item representation with the original item latent representation through residual connections and propagates neighborhood information on the user-item interaction graph, thereby incorporating user interaction preferences while preserving the multimodal features of the items themselves. Joint optimization of ranking loss and diffusion loss ensures that the representation learning process aligns with the recommendation objective, which is beneficial for improving the accuracy of recommendation results. Furthermore, this invention can employ a deterministic skip-step update method to reduce the number of iterations in the representation enhancement process, thereby reducing computational overhead and inference latency, and improving its applicability in large-scale recommendation scenarios.

[0056] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0058] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0059] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal recommendation method based on multi-source neighborhood conditional diffusion, characterized in that, The method includes: A noise-added representation for the corresponding diffusion time step is generated based on a preset noise scheduling strategy; The network is refined by combining the noisy representation, the multi-source neighborhood conditional representation, and the temporal embedding input conditional diffusion representation corresponding to the diffusion time step to obtain the predicted representation of the target item. The diffusion loss is then calculated based on the difference between the latent representation of the target item and the predicted representation. The latent representation of the target item is obtained by mapping the multimodal features of the target item. Based on the conditional diffusion representation refinement network and the multi-source neighborhood conditional representation of the target item, the latent representation of the target item is updated conditionally at multiple time steps to obtain the enhanced item representation; The enhanced item representation is added to the latent representation of the target item by residual addition to obtain the initial representation of the item node; The initial representation of the item node is input into the interaction graph with the user embedding vector to propagate neighborhood information, and the ranking loss is calculated based on the final user representation and the final item representation. The parameters of the conditional diffusion representation refinement network are updated based on the diffusion loss and the ranking loss to obtain the updated conditional diffusion representation refinement network. The updated conditional diffusion representation refines the network to obtain the final candidate item representation and the corresponding end-user representation, and performs an inner product to obtain a recommendation score. The item recommendation results are then obtained by sorting the items according to the recommendation scores.

2. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 1, characterized in that, Before generating the noisy representation of the corresponding diffusion time step according to the preset noise scheduling strategy, the method further includes: Obtain multimodal features of the item set and user-item interaction data, and construct an interaction graph based on the interaction data; A semantic neighborhood graph is constructed based on the similarity of item content features between different items in the same modality obtained from the multimodal features, and a behavioral neighborhood graph is constructed based on the behavioral association strength between different items obtained from the interaction data. Map the multimodal features of items in the item set to latent representations of the items; Extract the semantic neighborhood relation vector and the behavioral neighborhood relation vector corresponding to the target item in the item set based on the semantic neighborhood graph and the behavioral neighborhood graph; The semantic neighborhood relation vector and the behavioral neighborhood relation vector are weighted and fused to obtain a fused neighborhood relation vector; The multi-source neighborhood conditional representation of the target item is obtained by weighted aggregation based on the item's latent representation and the fused neighborhood relation vector.

3. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 2, characterized in that, The step of constructing a semantic neighborhood graph based on the similarity of item content features among different items in the same modality obtained from the multimodal features, and constructing a behavioral neighborhood graph based on the behavioral association strength among different items obtained from the interaction data, includes: Based on the multimodal features, extract the similarity of item content features between the target item and any other item in the item set under the same modality; For the target item, after sorting the similarity of the item content features in descending order, a preset number of item connections are retained and normalized to obtain the semantic neighborhood graph; The behavioral association strength between the target item and any other item in the item set is obtained based on the first set of interacting users corresponding to the target item and the second set of interacting users corresponding to any other item in the item set. For the target item, after sorting the behavior association strength in descending order, a preset number of item connections greater than a preset filtering threshold are retained and normalized to obtain the behavior neighborhood graph.

4. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 1, characterized in that, The step of performing conditional iterative updates on the latent representation of the target item at multiple time steps based on the conditional diffusion representation refinement network and the multi-source neighborhood conditional representation of the target item to obtain the enhanced item representation includes: Based on the conditional diffusion representation refinement network, the conditional prediction result and the unconditional prediction result are obtained by taking the multi-source neighborhood conditional representation and the empty conditional representation as inputs, respectively; wherein, the empty conditional representation is a zero vector with the same dimension as the multi-source neighborhood conditional representation. The conditional prediction result and the unconditional prediction result are linearly combined based on the conditional guidance intensity parameter to obtain the target representation prediction result. Based on the target representation prediction results, perform conditional iterative updates at multiple time steps to obtain the enhanced item representation.

5. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 4, characterized in that, The conditional iterative update that performs multiple time steps is represented as follows: ； in, Indicates the first The output state obtained after a deterministic conditional jump step update at each time step. Indicates the first The noise preservation factor corresponding to each time step. Indicates from the first time step to the second time step. The cumulative product of the noise preservation coefficients at each time step Indicates the first The target corresponding to each time step represents the prediction result. Indicates the first The target item in the modality The multi-source neighborhood condition representation.

6. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 4, characterized in that, Conditional iterative updates also include skip updates, the skip update process of which includes: From containing Select from the complete time step sequence of each time step that contains The target time step sequence at each time point; The target item in modality The potential representation of the item is set as the initial state of the representation in the target time step sequence; Based on the aforementioned representation, the initial state is refined, and a deterministic conditional step-by-step update is performed according to the target time step sequence. Will be completed The output state obtained after the deterministic conditional jump update is determined as the enhanced item representation.

7. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 1, characterized in that, The diffusion loss is expressed as follows: ； in, Representing modes The diffusion loss below, Indicates the quantity of the target item. Indicates the first The potential representation of an item corresponding to a target item. Indicates the first The target item is in the first The noise representation at each time step Indicates the first The target item in the modality The multi-source neighborhood condition representation under the following conditions Indicates the first The loss weights corresponding to each time step Indicates the first The prediction representation corresponding to each time step. This represents the 2-norm.

8. The multimodal recommendation method based on multi-source neighborhood conditional diffusion according to claim 7, characterized in that, The ranking loss is expressed as follows: ； in, Represents the sorting loss. Indicates training triplet The set that is formed This represents the Sigmoid function. Indicates transpose. Indicates to users Target items with interactive relationships The final item represents, Indicates to users Items that do not have an interaction relationship The final item represents, This represents the end-user's perspective.

Images