A multi-modal user-item recommendation method based on collaborative multi-granularity semantic alignment

Abstract

This invention provides a multimodal user-item recommendation method based on collaborative multi-granular semantic alignment, comprising: acquiring user historical interaction data and item multimodal information; extracting feature vectors of items in textual, visual, and collaborative modalities; constructing and fusing modality-specific item-item similarity graphs to generate a fused adjacency matrix; utilizing graph convolutional networks for representation learning and pre-alignment of user embeddings, item ID embeddings, and multimodal embeddings; maintaining semantic consistency between modalities and hierarchical similarity within modalities through hierarchical similarity comparison learning; and combining a lightweight attention diffusion model to resolve fine-grained conflicts between modalities, achieving local semantic alignment; and finally calculating preference scores based on the optimized user and item embeddings to generate personalized recommendation results, significantly improving multimodal recommendation accuracy and user experience.

Description

Technical Field

[0001] This invention belongs to the field of item recommendation technology, specifically relating to a multimodal user item recommendation method based on collaborative multi-granularity semantic alignment. Background Technology

[0002] With the rapid development of the internet and e-commerce, recommender systems have become core tools for information filtering and personalized services. Traditional recommender systems mainly rely on user-item interaction data, learning user preferences through methods such as collaborative filtering, achieving significant application results. However, with the increasing richness of multimodal item information such as text and images, multimodal recommendation has gradually become a research hotspot. By utilizing the visual, textual, and other modal information of items, multimodal recommendation can deeply mine item features and user preferences, thereby providing more accurate recommendation results in scenarios with sparse interaction data.

[0003] Existing multimodal recommendation methods mainly fall into two categories: one is feature-based fusion methods, which directly fuse multimodal features with item ID embeddings through simple concatenation or attention-weighted averaging, and then perform recommendations based on collaborative filtering. Although this method is simple to implement, it ignores the semantic differences between different modalities. Unaligned multimodal features may introduce noise and even degrade the overall performance of the recommendation model. The other category is structure-based graph methods, which construct modality-specific item-item graphs by calculating item feature similarity and use graph neural networks to learn the latent semantic relationships between items. For example, in existing technologies, CN119202398A discloses a multimodal recommendation method and system that enhances user representation using graph convolutional neural networks. This invention includes extracting item embeddings, original user embeddings, and original ID embeddings from the input dataset; extracting features from the user's neighbors and performing aggregation operations to enhance the user representation to generate user embeddings that aggregate other user information; concatenating these with the item embeddings to form a user-item collaborative interaction matrix; extracting features from each modality from the input dataset and constructing user-item graphs for each modality using the user-item collaborative interaction matrix; and then obtaining the user's recommended items through multimodal information representation, multimodal feature fusion, and recommendation results. Such methods can integrate cross-modal structural relationships between items into the collaborative learning process, enhancing the structural semantic information of item representations. To further align multimodal features of items with collaborative item ID embeddings, recent research has also introduced self-supervised contrastive learning mechanisms. By maximizing the semantic similarity of positive sample pairs and minimizing the similarity of negative samples, cross-modal semantic consistency and item discriminability are improved.

[0004] Despite the progress made by existing methods, significant limitations remain: 1. Insufficient semantic alignment from a holistic perspective: Existing methods typically process each modality independently when aggregating modality-specific graphs, making it difficult to eliminate semantic gaps between modalities. Furthermore, standard negative sampling strategies may disrupt the learned hierarchical similarity structure within a modality during subsequent contrastive learning, reducing the quality of representation learning.

[0005] 2. Semantic conflict at the local perspective: Subtle conflicts exist between different modalities. For example, the details of an object captured by the visual modality may differ from the textual description. While traditional contrastive learning can enhance overall semantic consistency, it cannot effectively resolve local inconsistencies of the same object across different modalities, which can impact representation learning and semantic alignment.

[0006] Therefore, there is an urgent need for a method that can collaboratively model multi-granularity semantic alignment, which can achieve semantic alignment between modalities at the overall level and reduce conflicts between modalities at the local level, thereby improving the accuracy and robustness of multimodal recommendations. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multimodal user item recommendation method based on collaborative multi-granularity semantic alignment.

[0008] The objective of this invention can be achieved through the following technical solutions: This invention provides a multimodal user item recommendation method based on collaborative multi-granularity semantic alignment, comprising the following steps: Acquire user-item interaction data and multimodal information about the items; The interaction data is preprocessed to obtain a user-item interaction matrix, and the multimodal information is processed to obtain modal feature vectors for each modality of the item. Based on the user-item interaction matrix and the modal feature vectors of each modality of the item, an item-item similarity graph of multiple modalities is constructed, and the attention weights of the graph structure are calculated by a graph neural network and fused to obtain the adjacency matrix of each modality. Based on the adjacency matrix after modal fusion, alignment embedding is performed through graph convolution and representation learning to obtain user embedding, pre-aligned item ID embedding, and multimodal embedding vector of the item. Based on the pre-aligned user embedding, item ID embedding, and multimodal embedding vectors, a total loss function based on BPR loss, contrastive learning loss, and diffusion loss is constructed. The user embedding, item ID embedding, and multimodal embedding vectors are then optimized according to the total loss function to obtain the optimized user embedding, item ID embedding, and item multimodal embedding vectors. Based on the optimized user embedding, item ID embedding, and item multimodal embedding vector, the user's preference score for the item is calculated, and the corresponding recommendation result for the user is generated according to the preference score.

[0009] Furthermore, the interaction data is the user's historical behavior data with items on the recommendation platform. The historical behavior data includes the number of times the user interacts with each item, and the interaction includes click behavior, browsing behavior, favorite behavior, add-to-cart behavior, and purchase behavior. The multimodal information includes visual modal information for characterizing the appearance information of an item and textual modal information for characterizing the semantic description information of an item. The visual modal information is extracted from the image data corresponding to the item, and the textual modal information is extracted from the textual description data corresponding to the item.

[0010] Furthermore, the preprocessing of the interaction data to obtain the user-item interaction matrix specifically includes: The interaction data is subjected to 5-core filtering, retaining items that have interacted with at least 5 or more users, and retaining users who have interacted with at least 5 or more items, resulting in a filtered user-item interaction matrix. This interaction matrix is ​​represented as follows: ,in, Indicates user With items There is interaction between them. Indicates user With items There is no interaction between them; For the set of users to be retained, A collection of items to be retained; , These represent the total number of users and the total number of items retained, respectively.

[0011] Furthermore, the processing of the multimodal information to obtain the modal feature vectors of each modality of the item specifically includes: The multimodal information is processed to extract the information for each item. In different modes Modal eigenvectors under ,in, For modality The embedding dimension, and for each modality Use a pre-trained model to extract feature vectors from image data and text description data of items; Indicates items In modality Modal eigenvectors under; The total number of modalities includes visual modalities and textual modalities. The visual modalities correspond to the image features of the items, which are extracted from the image data through a pre-trained convolutional neural network. The textual modalities correspond to the textual description features of the items, which are extracted from the descriptive text of the items through a pre-trained natural language processing model.

[0012] Furthermore, the process of constructing multi-modal item-item similarity graphs based on the modal feature vectors of each item modality, and then calculating the attention weights of the graph structure using a graph neural network to fuse them, thereby obtaining the adjacency matrix after modal fusion, specifically includes: Based on the modal feature vectors of the items Construct an item-item similarity graph with multiple modalities, where each modality... This corresponds to a similarity graph, where nodes represent items and edges represent the similarity between items. The similarity is calculated using the following formula: in, Represents items With items In modality Similarity; Indicates items In modality Modal eigenvectors under; It is not the L2 norm of a vector; For each modality, the item-item similarity graph is denoised by removing edges with similarity lower than the average similarity for that modality, and for each item node, only the one with the highest similarity is retained. Given adjacent item nodes, with edge weights set to 1, we obtain the pruned modal item-item similarity graph, whose adjacency matrix is ​​represented as follows: ,in, Indicating in modality m Lowering items With items There are similar connections between them. These are preset hyperparameters; Introducing collaborative modalities based on user-item interaction matrices The collaborative similarity relationship between items is calculated, and the number of users interacting with each other is used as the collaborative similarity between items to construct an item-item similarity graph in the collaborative modality. The collaborative similarity is calculated using the following formula: in, Represents items With items The collaborative similarity between them. Similarly, for each item node, only the one with the highest similarity is retained. Given adjacent item nodes, with edge weights set to 1, we obtain the pruned collaborative item-item similarity graph, whose adjacency matrix is ​​represented as follows: ; Based on the obtained adjacency matrices of each modality, including the visual modality, text modality, and the newly introduced collaborative modality, the structural similarity between different modalities is calculated. By counting the number of shared neighbors in the similarity graphs of different modalities, the structural similarity coefficient between modalities is obtained. in, This represents a statistical operation on the number of shared neighbors; Representing modes With mode The structural similarity coefficient between them; Indicates mode With mode The adjacency relationship is taken as the intersection, which is used to obtain the adjacency relationship of items that exist in both modes; Based on the structural similarity coefficients, the structural similarity coefficients corresponding to each mode are normalized to obtain the graph structure attention weights between modes: in, Representing modes For modes Attention weights in graph structure; Based on the attention weights of the graph structure, the adjacency matrices of each modality are weighted and fused to obtain the fused multimodal item-item similarity graph, whose adjacency matrix is ​​represented as follows: in, Representing modes The merged adjacency matrix These are preset weighting parameters used to balance the information of the original modal structure and the cross-modal structure.

[0013] Furthermore, based on the adjacency matrix after modal fusion, representation learning and embedding alignment are performed through graph convolution to obtain user embeddings, pre-aligned item ID embeddings, and multimodal embedding vectors of items, specifically including: Using LightGCN as the graph convolution kernel, leveraging =1 convolutional layer fusing the item-item image from each modality Perform convolution to obtain the multimodal embedding vector for each item. ; And apply to the user-item interaction matrix =2 convolutional layers to obtain the user embedding Embedded with the first item ID , Represents items Embedded representation in the interaction diagram; By all Perform average pooling to obtain the fused graph. and in the fusion diagram Embed the first item ID above Perform convolution to obtain the second item ID embedding. and embedded with the first item ID before convolution. Perform residual joins to generate pre-aligned item ID embeddings. The formula is: in, This is a hyperparameter.

[0014] Furthermore, the total loss function is expressed as: in, This is the total loss function; The BPR loss is used to optimize the prediction of user preferences and items, ensuring that the items generated by the recommendation system match the user's interests. The contrastive learning loss is used to optimize the similarity between item ID embeddings and item multimodal embeddings through contrastive learning; , These are the diffusion embedding recovery loss and the diffusion embedding alignment loss, respectively, used to optimize the ability of the diffusion model to recover and align the embeddings. , Preset weighting coefficients; For L2 norm regularization terms; The BPR loss is expressed as: in, This represents a triple sampled from the user-item interaction matrix, where user With items There is an interactive relationship, and the user With items There is no interaction relationship; Indicates user User embedding; Represents items The final representation vector, based on the item The multimodal embedding vectors are fused together to obtain the results.

[0015] Furthermore, the contrastive learning loss is expressed as: in, To compare learning loss, , Preset weighting coefficients; For basic contrastive learning loss; To contrast the learning loss in a hierarchical manner; The batch size represents the number of items used in a single training iteration during contrastive learning. Represents items ID embedding vectors and items modality m Similarity function between embedded vectors; For temperature parameters; Indicates cosine similarity; It represents a set of modalities, including visual modalities, textual modalities, and collaborative modalities; Represents items Pre-aligned item ID embedding; Represents items In modality The multimodal embedding vector below; Represents items The sum of similarity scores across all modalities is greater than the preset hyperparameters. A multimodal set of similar items; Represents items The sum of similarity scores across all modalities is less than the hyperparameter. However, the set of single-modal similar items that is greater than or equal to the preset value of 2; Represents items The set of dissimilar items whose sum of similarity scores across all modalities is less than a preset value of 2.

[0016] Furthermore, the diffusion embedding recovery loss is expressed as: in, This represents the item's representation within the initial item ID embedding space; This represents the optimized item ID embedding representation. This represents the mean squared error value of item ID embedding before and after diffusion. The diffusion embedding alignment loss is expressed as: in, Represents the representation of an item in the initial multimodal embedding space; This represents the optimized multimodal embedding representation. This represents the mean squared error of the multimodal embedding of the item before and after diffusion.

[0017] Furthermore, the step of calculating the user's preference score for items based on the optimized user embedding, item ID embedding, and item multimodal embedding vector, and generating a recommendation result for the corresponding user based on the preference score, specifically includes: Based on the optimized item ID embedding and item multimodal embedding vector, the final item representation is calculated using the following formula: in, Represents items The final representation vector, For optimized item ID embedding, Items in the multimodal embedding vectors of items Optimized embedding vectors in textual, visual, and collaborative modalities. This indicates the calculation of the average value; Based on the final item representation and the optimized user embedding, the user's preference score for each item is calculated using the following formula: in, Indicates user Optimized user embedding; T For transpose; Indicates user For items Preference scores; Items are sorted according to preference scores, and the item with the highest score is selected as the recommendation result to generate a personalized recommendation list for users to view.

[0018] Compared with the prior art, the present invention has the following advantages: (1) Existing recommendation methods often neglect the collaborative modeling of intramodal similarity and intermodal alignment. Specifically, during the construction of modality-specific graphs, the similarity graphs of each modality are constructed independently, leading to semantic differences and gaps between modalities, which in turn affects the performance of the recommendation system. To overcome this problem, this invention constructs item-item similarity graphs for multiple modalities and uses graph neural networks to calculate the attention weights of the graph structures for weighted fusion between modalities. In this way, the semantic differences between multiple modalities can be effectively aligned, ensuring that information from different modalities can work collaboratively, thus improving the performance of the recommendation model. In summary, this invention optimizes the semantic alignment between modalities through weighted fusion of graph structure attention between modalities, thereby improving the accuracy and robustness of the recommendation results.

[0019] (2) Existing contrastive learning methods typically employ standard negative sampling strategies, which may disrupt the learned hierarchical similarity structure within modalities, leading to information loss and increased noise during the learning process. Therefore, existing methods have certain limitations in the contrastive learning stage. This invention introduces a multi-granularity semantic alignment strategy to fine-grainedly optimize the comparison between different modalities in the contrastive learning loss. This method strengthens the semantic consistency between modalities by comparing item ID embeddings with item multimodal embeddings, while preserving intramodal differences from a global perspective, thus avoiding the disruption of intramodal similarity structures caused by negative sampling strategies. This method effectively improves the learning quality and stability of the recommendation model and reduces information loss due to insufficient similarity comparison.

[0020] (3) Existing technologies have problems with semantic alignment from a local perspective, especially in handling subtle conflicts between different modalities. Different modalities (such as visual modalities and textual modalities) may provide different detailed descriptions of the same item. Such subtle conflicts cannot be effectively handled by traditional contrastive learning methods, thus affecting the accuracy of recommendation results. This invention introduces a diffusion model to guide the diffusion of subtle conflicts between modalities, aligning them after a few iterations. By correcting conflicts through diffusion, this invention solves the inconsistency problem between modalities from a local perspective and improves the alignment effect of item features. Through this innovation, this invention effectively enhances the overall consistency of multimodal representations, improving the accuracy of recommendation results and user experience.

[0021] (4) Most existing methods rely on direct splicing or simple weighted fusion of multimodal features. While convenient, these methods fail to adequately consider the contributions of different modal information at different levels, easily leading to information redundancy or noise interference, thus affecting recommendation accuracy. This invention designs an embedding alignment mechanism based on graph convolution for representation learning, combining multimodal similarity graphs and user-item interaction matrices to learn and ultimately fuse information in association across different modalities. By using a graph convolutional network (GCN) to learn the fused item similarity graph, the final representation of each item not only includes the fusion of multimodal information but also optimizes user embedding and item ID embedding through collaborative learning. This mechanism effectively avoids the drawbacks of noise interference between modal information in traditional methods, improving the diversity and accuracy of recommendations. Attached Figure Description

[0022] Figure 1 This is a flowchart of the multimodal user item recommendation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the model structure constructed in step S3 of an embodiment of the present invention; Figure 3 This is a schematic diagram of the model structure constructed in step S51 of an embodiment of the present invention; Figure 4 This is a schematic diagram of the model structure constructed in step S52 of this embodiment of the invention; Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] Example 1: This embodiment provides a multimodal user item recommendation method based on collaborative multi-granularity semantic alignment. This method improves the semantic consistency of item representations through global and local semantic alignment, such as... Figure 1 As shown, the specific steps include: Step S1: Obtain interaction data between the user and the item, and multimodal information about the item; Acquire user-item interaction data and corresponding multimodal information about the items. This interaction data originates from the actual operation of the recommendation platform, reflecting users' historical behavior towards items and used to characterize their true interests and preferences. This historical behavior data may include, but is not limited to, user clicks, browsing, favorites, adding to cart, and purchases. Different types of interactions reflect the user's level of attention and preference intensity towards items from different perspectives. In practical applications, these various interactions can be uniformly mapped to implicit feedback signals, or different behaviors can be assigned different weights according to business needs, thereby enhancing the accuracy of user preference modeling.

[0025] Simultaneously, multimodal information associated with items is acquired to enrich item representations at the content level. This multimodal information includes at least visual and textual modal information. Visual modal information characterizes the appearance features of items and originates from corresponding image data, such as product images or display images. Textual modal information characterizes the semantic description features of items and originates from the item's title, attribute descriptions, or detailed description text. In the implementation process, image data and textual description data can be preprocessed separately to ensure data quality and consistency, providing reliable input for subsequent feature extraction and multimodal modeling. By simultaneously introducing interaction data and multimodal information, a foundation can be laid for subsequent multimodal recommendation modeling from both collaborative and content signal levels, thereby alleviating the interaction sparsity problem and improving recommendation performance.

[0026] Step S2: Preprocess the interaction data to obtain the user-item interaction matrix, and process the multimodal information to obtain the modal feature vectors of each modality of the item, specifically including: Organize and filter historical interaction data between users and items. Let the user set be... Include Individual users, collection of items Includes | | Items. To reduce the impact of noisy interactions on the training stability of the recommendation model and alleviate the problem of extremely sparse data, a 5-core filtering strategy is applied to the original interaction data. This strategy retains only users who have interacted with at least 5 different items, and only items that have been interacted with by at least 5 different users. This filtering process yields the selected user set. and item collection .

[0027] Based on the filtered interaction data, construct a user-item interaction matrix. The interaction matrix is ​​represented as: in, Indicates user With items There is interaction between them. Indicates user With items There is no interaction between them; For the set of users to be retained, A collection of items to be retained; , These represent the total number of users and the total number of items retained, respectively. This interaction matrix is ​​used to characterize the implicit preference signals of users and provides basic constraints for subsequent embedding learning based on collaborative filtering.

[0028] Subsequently, the multimodal information of the items is processed and features are extracted. Let the modality set be... ={t,v}, where t represents the text modality and v represents the visual modality. In this embodiment, features of different modalities are extracted by corresponding pre-trained models to ensure that the modal representations have strong semantic expressive power and generalization ability.

[0029] For the text modality, for each item i The text description data is used to extract semantic features using a pre-trained natural language processing model to obtain text modality feature vectors. Text modality features are used to characterize the semantic attributes, functional descriptions and usage scenarios of items, which helps to depict users' semantic interests and preferences.

[0030] For visual modalities, for each item i The corresponding image data is used to extract its appearance features using a pre-trained convolutional neural network or visual coding model to obtain a visual modality feature vector. Visual modality features can reflect the appearance style, color distribution and structural information of the item, which helps to improve the consistency of the recommendation results at the perception level.

[0031] Finally obtain each item In different modes Modal eigenvectors under ,in, For modality The embedding dimension.

[0032] Step S3: Based on the user-item interaction matrix and the modal feature vectors of each modality of the item, construct item-item similarity graphs for multiple modalities, and calculate the attention weights of the graph structure using a graph neural network, then fuse them to obtain the fused adjacency matrix of each modality, such as... Figure 2 As shown, this step establishes multiple graph structures and merges them to create connections, enabling pre-alignment of encoded features. The specific implementation is as follows: S31: Construct a modality-specific item-item similarity graph for each modality. For any modality ∈ Based on the feature vectors of items in this modality, the cosine similarity between items is calculated to form a modality similarity matrix. The calculation method for any element in the matrix is ​​as follows: in, Represents items With items In modality Similarity; Indicates items In modality Modal eigenvectors under; It is the L2 norm of the vector; cosine similarity can eliminate the difference in vector scale and only focus on the similarity of different items in the semantic direction.

[0033] S32: Denoise the similarity matrix obtained for each modality. Specifically, in each modality, only the top k most similar neighbors of each item are retained (k is a hyperparameter, such as k=10; in this embodiment, k=10). Simultaneously, edges with significantly lower-than-average similarity in other modalities are removed to reduce the impact of accidental similarity and noisy connections. After the above processing, the sparse adjacency matrix corresponding to each modality is obtained. This adjacency matrix can characterize relatively reliable item similarity relationships within a mode.

[0034] S33: Introducing collaborative modalities to supplement the structural information implicit in user behavior. Collaborative modalities characterize the collaborative relationships between items through user-item interaction data. Specifically, based on the user-item interaction matrix, an item similarity matrix is ​​constructed under the collaborative modalities, where each element represents the number of users who have interacted with two items simultaneously, calculated as follows: in, Represents items With items The collaborative similarity between items is calculated. To avoid noise from accidental clicks, a filtering operation is also performed on the collaborative similarity matrix, retaining only edges with at least a threshold ξ=2 shared users, and for each item, only the top k most relevant neighbors are retained, thus constructing the adjacency matrix corresponding to the collaborative mode. .

[0035] S34: Graph Structure Fusion: For graph structures of different modalities, the common edges of multiple graphs have stronger connections than the edges existing in a single graph. Based on this, this embodiment first calculates the number of shared edges as the structural similarity coefficient. and Calculate the similarity coefficient: in, This represents a statistical operation on the number of shared neighbors; Representing modes With mode The structural similarity coefficient between them; Indicates mode With mode The adjacency relationships are taken as the intersection to obtain the adjacency relationships of items that exist in both modalities; then the structural similarity coefficients are normalized along the modalities to obtain the structural attention weights: in, Representing modes For modes Attention weights in graph structure; The adjacency matrix of all modalities is then weighted and fused based on this weight. The final fused matrix... The calculation formula is: in, Representing modes The merged adjacency matrix These are preset weight parameters used to balance the original modal structure and cross-modal structural information. Through this fusion process, the relationships between items in different modalities are mapped to a unified structural space, enabling preliminary semantic alignment of multimodal features at the graph structure level, thus laying the foundation for subsequent graph convolutional propagation and multi-granularity semantic alignment learning.

[0036] Step S4: Based on the adjacency matrix after modal fusion, perform representation learning and alignment embedding through graph convolution to obtain user embedding, pre-aligned item ID embedding, and multimodal embedding vector of the item. This step jointly models embeddings from different sources under a unified graph structure constraint, enabling explicit connections between collaborative and multimodal representations of items during the propagation phase, thereby achieving preliminary semantic alignment at the structural level.

[0037] In one specific embodiment, step S4 is implemented as follows: For each modality, a graph convolution operation is performed on the fused item-item graph to learn the contextual representation of the item in that modality. The graph convolution uses a lightweight LightGCN as the kernel, retaining only the neighbor aggregation process without introducing nonlinear transformations and feature transformation matrices, thus reducing parameter size and highlighting the structure propagation effect. For each modality... m For the fused adjacency matrix Using a single-layer graph convolution ( =1), obtain the item in modality m Embedded vectors ; Learn user embeddings and item ID embeddings on the user-item interaction graph. Construct a user-item bipartite graph based on the user-item interaction matrix obtained in step S2, and perform two-layer graph convolution on this graph (convolution layer number...). =2), to capture higher-order collaboration relationships. After graph convolution, the user embedding vector is obtained. And the first item ID embedded in the interaction graph This embedding is driven solely by user behavior data and reflects collaborative structure information based on user group preferences.

[0038] By all Perform average pooling to obtain the fused graph. and in the fusion diagram Embed the first item ID above Perform convolution to obtain the second item ID embedding. and embedded with the first item ID before convolution. Perform residual joins to generate pre-aligned item ID embeddings. The formula is: in, This is a hyperparameter.

[0039] Step S5: Based on the pre-aligned user embedding, item ID embedding, and multimodal embedding vectors, construct a total loss function based on BPR loss, contrastive learning loss, and diffusion loss, and optimize the user embedding, item ID embedding, and multimodal embedding vectors according to the total loss function to obtain the optimized user embedding, item ID embedding, and item multimodal embedding vectors. The process of constructing the loss function in this step specifically includes: Step S51: In this embodiment, by introducing a hierarchical similarity comparison mechanism, while maintaining intramodal similarity structures, overall semantic alignment between item ID embedding and multimodal embedding is achieved, thereby alleviating the semantic misalignment problem between different modalities and improving the discriminative ability and stability of the representation: such as Figure 3 As shown, this step achieves overall semantic alignment through contrastive learning while maintaining intra-modal hierarchical similarity. The specific implementation is as follows: S511: Basic Comparative Learning: Embedding with Item ID Anchor point, corresponding to modal representation For positive samples, the modal representation of other items within the batch. For negative samples, the InfoNCE loss function is used for aligned item ID embedding and multimodal embedding: in = , For calculation and The cosine similarity, where τ is a temperature parameter (e.g., τ=0.2). Indicates the negative sample index. It refers to the batch size.

[0040] S512: Hierarchical contrastive learning: Treating item ID embedding as an additional modality ( ∪{id}), and calculate each mode Similarity matrix ,in Define a strong similarity threshold And weak similarity threshold ( This represents the k-th largest similarity value. (This represents the 2kth largest similarity value). Similarity scores between items are calculated based on strong and weak similarity thresholds: Reconstruct the aggregated similarity rating matrix And based on the threshold (Hyperparameters) categorize items into multimodal similarity levels. (when ≥ ,), Single-modal similarity (when ≥2) and dissimilar (when <2), and then calculate the hierarchical contrast loss: Finally, through hyperparameters The ratio of managing the two losses: in, To compare learning loss, , Preset weighting coefficients; For basic contrastive learning loss; To contrast the learning loss in a hierarchical manner; The batch size represents the number of items used in a single training iteration during contrastive learning. Represents items ID embedding vectors and items modality m Similarity function between embedded vectors; For temperature parameters; Indicates cosine similarity; It represents a set of modalities, including visual modalities, textual modalities, and collaborative modalities; Represents items Pre-aligned item ID embedding; Represents items In modality The multimodal embedding vector below; Represents items The sum of similarity scores across all modalities is greater than the preset hyperparameters. A multimodal set of similar items; Represents items The sum of similarity scores across all modalities is less than the hyperparameter. However, the set of single-modal similar items that is greater than or equal to the preset value of 2; Represents items The set of dissimilar items whose sum of similarity scores across all modalities is less than a preset value of 2.

[0041] Step S52: Construct a lightweight attentional diffusion model to achieve fine-grained semantic alignment of modalities; In this embodiment, a lightweight attentional diffusion model is introduced, which alleviates the semantic conflict problem between different modalities at the local detail level by performing a low-step diffusion and denoising process in the multimodal embedding space, thereby further improving fine-grained semantic consistency on the basis of overall contrastive learning alignment. Figure 4 As shown, this step uses a lightweight diffusion model to resolve modal detail conflicts, enabling fine-grained semantic alignment. The specific implementation is as follows: S521: Define the degradation operator: Based on the cold diffusion framework, use the degradation operator. First, Gaussian noise is used. Modal embedding generates semantic mixed noise During the forward diffusion phase, through the degenerate operator Injecting time step t noise into multimodal features: in Control the noise scheduling and adjust it through linear scheduling: s is the noise ratio hyperparameter. It is a noise-free modal embedding at the beginning.

[0042] S522: Define the recovery operator: Based on the cold diffusion framework, the recovery operator is used in the back diffusion stage. To remove noise, we obtain: in The noise is predicted by a semantic conflict estimator. The semantic conflict estimator uses a single-layer, single-head TransformerEncoder as its core, identifies semantically conflicting embeddings through attention weights, and then identifies high-confidence modalities through gating signals. This is done to guide the diffusion of conflicting modalities towards it in order to achieve semantic alignment. For training the semantic conflict estimator, this embodiment uses: in, This represents the item's representation within the initial item ID embedding space; This represents the optimized item ID embedding representation. This indicates the calculation of the mean squared error value for item ID embedding; Represents the representation of an item in the initial multimodal embedding space; This represents the optimized multimodal embedding representation. This indicates the calculation of the mean square error value of multimodal embedding of an item.

[0043] Its ability to predict noise and its ability to align semantics are trained separately, specifically, through The ability of a trained model to predict noise in order to recover the original features. The model is trained to handle detailed conflicts in order to achieve alignment at the local granular level.

[0044] S523: Local fine-grained semantic alignment: During the inference phase, this embodiment assumes features (All modal splicing) already contains noise at time step T, i.e., it is assumed = , directly to embed Iterative application of recovery operator From t=T to t=1: Finally, the original information is preserved through residual join: Where μ is a hyperparameter, Representing different modal embeddings, this step achieves local alignment with low computational overhead (only 1-4 steps of diffusion), and works in conjunction with contrastive learning to improve multi-granular semantic consistency.

[0045] Step S53: This step integrates all loss functions, and the total loss function is expressed as: in, This is the total loss function; The BPR loss is used to optimize the prediction of user preferences and items, ensuring that the items generated by the recommendation system match the user's interests. The contrastive learning loss is used to optimize the similarity between item ID embeddings and item multimodal embeddings through contrastive learning; , These are the diffusion embedding recovery loss and the diffusion embedding alignment loss, respectively, used to optimize the ability of the diffusion model to recover and align the embeddings. , Preset weighting coefficients; For L2 norm regularization terms; BPR loss is expressed as: in, This represents a triple sampled from the user-item interaction matrix, where user With items There is an interactive relationship, and the user With items There is no interaction relationship; Indicates user User embedding; Represents items The final representation vector, based on the item The multimodal embedding vectors are fused together to obtain the results.

[0046] Step S6: Based on the optimized user embedding, item ID embedding, and item multimodal embedding vector, calculate the user's preference score for the item, and generate the corresponding recommendation result for the user based on the preference score, specifically including: Based on the optimized item ID embedding and item multimodal embedding vector, the final item representation is calculated using the following formula: in, Represents items The final representation vector, For optimized item ID embedding, Items in the multimodal embedding vectors of items Optimized embedding vectors in textual, visual, and collaborative modalities. This indicates the calculation of the average value; Based on the final item representation and the optimized user embedding, the user's preference score for each item is calculated using the following formula: in, Indicates user Optimized user embedding; T For transpose; Indicates user For items Preference scores; Items are sorted according to preference scores, and the item with the highest score is selected as the recommendation result to generate a personalized recommendation list for users to view.

[0047] As described above, this invention presents a multimodal recommendation method based on collaborative multi-granularity semantic alignment, aiming to deeply utilize information from different modalities of items to improve recommendation accuracy. Addressing the shortcomings of previous recommendation methods, this invention proposes a pre-aligned graph encoder module and a hierarchical similarity comparison mechanism to collaboratively model intermodal alignment and intramodal similarity. The former initially aligns modal features by fusing graph structures to alleviate subsequent alignment pressure, while the latter maintains the hierarchical similarity structure learned by the former through comparative differentiation while aligning semantics overall. Furthermore, this invention proposes a lightweight attention diffusion module, which uses gating and attention mechanisms to identify and mitigate detailed conflicts between modalities to achieve local semantic alignment, and is co-designed with comparative alignment to improve computational efficiency. This method fully utilizes information from different modalities and performs excellently in sparse and noisy scenarios, thus possessing practical application prospects.

[0048] This invention collaboratively models the alignment between object modalities and the similarity within modalities. In the feature learning stage, a pre-aligned graph encoder initially aligns the features of different object modalities, reducing the burden of subsequent alignment. In the contrastive alignment stage, while aligning the features of different modalities overall, the hierarchical similarity structure learned in the previous stage is maintained through comparative differentiation. Furthermore, it alleviates detail conflicts between modalities. A lightweight attention diffusion module aligns the detailed features of different object modalities at the local level, reducing semantic gaps and noise impact. Through the collaborative design with contrastive alignment, computational overhead is reduced.

[0049] Example 2: This embodiment provides a multimodal user item recommendation system based on collaborative multi-granularity semantic alignment, including: Data Acquisition Module: This module collects interaction data between users and items, as well as multimodal information about the items. Interaction data includes historical user behaviors on the recommendation platform, such as clicks, browsing, favorites, adding to cart, and purchases. Each interaction record includes the user ID, item ID, and interaction type. Multimodal information includes visual and textual modal information. Visual modality is extracted from the image data corresponding to the item, and textual modality is extracted from the text description data corresponding to the item, used to characterize the item's appearance and semantic features.

[0050] The data preprocessing and feature extraction module filters user-item interaction data, removing users and items with insufficient interaction to obtain a user-item interaction matrix with reasonable sparsity. It also processes multimodal information, extracting feature vectors for each item in different modalities. Visual modalities are extracted using a pre-trained convolutional neural network, and textual modalities are extracted using a pre-trained natural language processing model, thus obtaining multimodal features that can represent the items.

[0051] The modality-specific graph construction and fusion module constructs a modality-specific item-item graph based on the item feature vectors of each modality, retaining the most similar items for each node and removing low-similarity edges. A collaborative modality graph is introduced, constructing a collaborative relationship graph between items based on user-item interaction data, retaining highly relevant item nodes. A graph structure attention mechanism is used to weight and fuse the adjacency matrices of each modality, forming a fused multimodal adjacency matrix to capture cross-modal structural information and achieve pre-alignment between modalities.

[0052] Embedding Alignment Module: This module generates user embeddings, item ID embeddings, and multimodal embeddings through graph convolution and representation learning. Convolution is performed on the item multimodal graph to generate item multimodal embeddings, and convolution is performed on the user-item interaction graph to generate user embeddings and initial item ID embeddings. Pre-aligned item ID embeddings are generated by fusing graph convolution and residual connections, achieving initial alignment between user and item embeddings.

[0053] The multi-granularity semantic alignment module includes a hierarchical similarity comparison module and a lightweight attention diffusion module. The hierarchical similarity comparison module performs overall semantic alignment between ID embeddings and multimodal embeddings through contrastive learning, while maintaining hierarchical similarity within each modality. The lightweight attention diffusion module introduces noise and recovery mechanisms to iteratively correct multimodal embeddings with local conflicts, achieving fine-grained semantic alignment. Furthermore, it uses a gating mechanism to guide conflicting modalities towards a consistent semantic direction, thereby improving the problem of local semantic conflicts.

[0054] Total Loss Optimization Module: Constructs a total loss function by combining BPR loss, contrastive learning loss, diffusion loss, and regularization term. Based on the total loss, it jointly optimizes user embedding, item ID embedding, and multimodal embedding to achieve collaborative optimization of preference prediction and semantic alignment.

[0055] Recommendation generation module: Based on the optimized user embedding and the final fused item representation, calculate the user's preference score for each item, and generate a personalized recommendation list based on the preference score for the user.

[0056] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal user item recommendation method based on collaborative multi-granularity semantic alignment, characterized in that, Includes the following steps: Acquire user-item interaction data and multimodal information about the items; The interaction data is preprocessed to obtain a user-item interaction matrix, and the multimodal information is processed to obtain modal feature vectors for each modality of the item. Based on the user-item interaction matrix and the modal feature vectors of each modality of the item, an item-item similarity graph of multiple modalities is constructed, and the attention weights of the graph structure are calculated by a graph neural network and fused to obtain the adjacency matrix of each modality. Based on the adjacency matrix after modal fusion, representation learning and alignment embedding are performed through graph convolution to obtain user embedding, pre-aligned item ID embedding, and multimodal embedding vector of the item. Based on the pre-aligned user embedding, item ID embedding, and multimodal embedding vectors, a total loss function based on BPR loss, contrastive learning loss, and diffusion loss is constructed. The user embedding, item ID embedding, and multimodal embedding vectors are then optimized according to the total loss function to obtain the optimized user embedding, item ID embedding, and item multimodal embedding vectors. Based on the optimized user embedding, item ID embedding, and item multimodal embedding vector, the user's preference score for the item is calculated, and the corresponding recommendation result for the user is generated according to the preference score.

2. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The interaction data refers to the historical behavior data of users interacting with items on the recommendation platform. The historical behavior data includes the number of times users interact with each item, and the interactions include click behavior, browsing behavior, favorite behavior, add-to-cart behavior, and purchase behavior. The multimodal information includes visual modal information for characterizing the appearance information of an item and textual modal information for characterizing the semantic description information of an item. The visual modal information is extracted from the image data corresponding to the item, and the textual modal information is extracted from the textual description data corresponding to the item.

3. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The preprocessing of the interaction data to obtain the user-item interaction matrix specifically includes: The interaction data is subjected to 5-core filtering, retaining items that have interacted with at least 5 or more users, and retaining users who have interacted with at least 5 or more items, resulting in a filtered user-item interaction matrix. This interaction matrix is ​​represented as follows: ,in, Indicates user With items There is interaction between them. Indicates user With items There is no interaction between them; For the set of users to be retained, A collection of items to be retained; , These represent the total number of users and the total number of items retained, respectively.

4. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The process of processing the multimodal information to obtain the modal feature vectors of each modality of the item specifically includes: The multimodal information is processed to extract the information for each item. In different modes Modal eigenvectors under ,in, For modality The embedding dimension, and for each modality Use a pre-trained model to extract feature vectors from image data and text description data of items; Indicates items In modality Modal eigenvectors under; The total number of modalities includes visual modalities and textual modalities. The visual modalities correspond to the image features of the items, which are extracted from the image data through a pre-trained convolutional neural network. The textual modalities correspond to the textual description features of the items, which are extracted from the descriptive text of the items through a pre-trained natural language processing model.

5. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The process involves constructing a multi-modal item-item similarity graph based on the user-item interaction matrix and the modal feature vectors of each item modality. Attention weights for the graph structure are then calculated using a graph neural network and fused to obtain the adjacency matrix after modal fusion. Specifically, this includes: Based on the modal feature vectors of the items Construct an item-item similarity graph with multiple modalities, where each modality... This corresponds to a similarity graph, where nodes represent items and edges represent the similarity between items. The similarity is calculated using the following formula: in, Represents items With items In modality Similarity; Indicates items In modality Modal eigenvectors under; It is the L2 norm of the vector; For each modality, the item-item similarity graph is denoised by removing edges with similarity lower than the average similarity for that modality, and for each item node, only the one with the highest similarity is retained. Given adjacent item nodes, with edge weights set to 1, we obtain the pruned modal item-item similarity graph, whose adjacency matrix is ​​represented as follows: ,in, Indicating in modality m Lowering items With items There are similar connections between them. These are preset hyperparameters; Introducing collaborative modalities based on user-item interaction matrices Calculate the collaborative similarity relationships between items, and Using the number of users interacting with each other as the collaborative similarity between items, an item-item similarity graph is constructed under the collaborative modality. The collaborative similarity is calculated using the following formula: in, Represents items With items The collaborative similarity between them. Similarly, for each item node, only the one with the highest similarity is retained. Given adjacent item nodes, with edge weights set to 1, we obtain the pruned collaborative item-item similarity graph, whose adjacency matrix is ​​represented as follows: ; Based on the obtained adjacency matrices of each modality, including the visual modality, text modality, and the newly introduced collaborative modality, the structural similarity between different modalities is calculated. By counting the number of shared neighbors in the similarity graphs of different modalities, the structural similarity coefficient between modalities is obtained. in, This represents a statistical operation on the number of shared neighbors; Representing modes With mode The structural similarity coefficient between them; Indicates mode With mode The adjacency relationship is taken as the intersection, which is used to obtain the adjacency relationship of items that exist in both modes; Based on the structural similarity coefficients, the structural similarity coefficients corresponding to each mode are normalized to obtain the graph structure attention weights between modes: in, Representing modes For modes Attention weights in graph structure; Based on the attention weights of the graph structure, the adjacency matrices of each modality are weighted and fused to obtain the fused multimodal item-item similarity graph, whose adjacency matrix is ​​represented as follows: in, Representing modes The merged adjacency matrix These are preset weighting parameters used to balance the information of the original modal structure and the cross-modal structure.

6. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 5, characterized in that, Based on the adjacency matrix after modal fusion, representation learning and embedding alignment are performed through graph convolution to obtain user embeddings, pre-aligned item ID embeddings, and multimodal item embedding vectors, specifically including: Using LightGCN as the graph convolution kernel, leveraging =1 convolutional layer fusing the item-item image from each modality Perform convolution to obtain the multimodal embedding vector for each item. ; And apply to the user-item interaction matrix =2 convolutional layers to obtain the user embedding Embedded with the first item ID , Represents items Embedded representation in the interaction diagram; By all Perform average pooling to obtain the fused graph. and in the fusion diagram Embed the first item ID above Perform convolution to obtain the second item ID embedding. and embedded with the first item ID before convolution. Perform residual joins to generate pre-aligned item ID embeddings. The formula is: in, This is a hyperparameter.

7. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The total loss function is expressed as: in, This is the total loss function; The BPR loss is used to optimize the prediction of user preferences and items, ensuring that the items generated by the recommendation system match the user's interests. The contrastive learning loss is used to optimize the similarity between item ID embeddings and item multimodal embeddings through contrastive learning; , These are the diffusion embedding recovery loss and the diffusion embedding alignment loss, respectively, used to optimize the ability of the diffusion model to recover and align the embeddings. , Preset weighting coefficients; For L2 norm regularization terms; The BPR loss is expressed as: in, This represents a triple sampled from the user-item interaction matrix, where user With items There is an interactive relationship, and the user With items There is no interaction relationship; Indicates user User embedding; Represents items The final representation vector, based on the item The multimodal embedding vectors are fused together to obtain the results.

8. The multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 7, characterized in that, The contrastive learning loss is expressed as: in, To compare learning loss, , Preset weighting coefficients; For basic contrastive learning loss; To contrast the learning loss in a hierarchical manner; The batch size represents the number of items used in a single training iteration during contrastive learning. Represents items ID embedding vectors and items modality m Similarity function between embedded vectors; For temperature parameters; Indicates cosine similarity; It represents a set of modalities, including visual modalities, textual modalities, and collaborative modalities; Represents items Pre-aligned item ID embedding; Represents items In modality The multimodal embedding vector below; Represents items The sum of similarity scores across all modalities is greater than the preset hyperparameters. A multimodal set of similar items; Represents items The sum of similarity scores across all modalities is less than the hyperparameter. However, the set of single-modal similar items that is greater than or equal to the preset value of 2; Represents items The set of dissimilar items whose sum of similarity scores across all modalities is less than a preset value of 2.

9. A multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 7, characterized in that, The diffusion embedding recovery loss is expressed as: in, This represents the item's representation within the initial item ID embedding space; This represents the optimized item ID embedding representation. This represents the mean squared error value of item ID embedding before and after optimization; The diffusion embedding alignment loss is expressed as: in, Represents the representation of an item in the initial multimodal embedding space; This represents the optimized multimodal embedding representation. This represents the mean square error of the multimodal embedding of items before and after optimization.

10. A multimodal user item recommendation method based on collaborative multi-granularity semantic alignment according to claim 1, characterized in that, The process of calculating a user's preference score for an item based on the optimized user embedding, item ID embedding, and item multimodal embedding vector, and generating a recommendation result for the corresponding user based on the preference score, specifically includes: Based on the optimized item ID embedding and item multimodal embedding vector, the final item representation is calculated using the following formula: in, Represents items The final representation vector, For optimized item ID embedding, Items in the multimodal embedding vectors of each item Optimized embedding vectors in textual, visual, and collaborative modalities. This indicates the calculation of the average value; Based on the final item representation and the optimized user embedding, the user's preference score for each item is calculated using the following formula: in, Indicates user Optimized user embedding; T For transpose; Indicates user For items Preference scores; Items are sorted according to preference scores, and the item with the highest score is selected as the recommendation result to generate a personalized recommendation list for users to view.

Images