A multimodal recommendation method and system based on multi-perspective cross-modal semantic alignment

Abstract

The present invention discloses a multimodal recommendation method and system based on multi-perspective cross-modal semantic alignment, and relates to the technical field of multimodal recommendation. The technical points of the present invention include: extracting user information, item information, user-item interaction information and multimodal auxiliary information of training data, wherein the multimodal auxiliary information includes textual modality and visual modality; processing the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation; inputting user information, item information, user-item interaction information and semantic information into a multimodal neural network model for training; and performing multimodal recommendation using the trained multimodal neural network model. The present invention solves the problem of semantic gap between modalities, and performs fine-grained modal semantic alignment through multi-perspective semantic modeling. Experimental results prove the superiority and effectiveness of the present invention.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal recommendation, and in particular to a multimodal recommendation method and system based on multi-perspective cross-modal semantic alignment. Background Art

[0002] Recommendation systems play an extremely important role in today's world. Predicting user preferences for items on various content consumption platforms such as e-commerce websites, social media websites, and portals can greatly improve the utilization rate of websites and also promote the improvement of user experience. At the same time, with the continuous development of the Internet, the use of various multimedia information including text, pictures, videos, audio, etc. can also attract target users to varying degrees. Different users can show different preferences and make different interactive behaviors through various multimedia information. Using this kind of multimodal auxiliary information to promote the recommendation performance of the recommendation system, this kind of recommendation task is collectively referred to as multimodal recommendation.

[0003] Using multimodal auxiliary information as supplementary knowledge to improve the performance of recommendation systems has shown good prospects and has been widely used. In order to explore the multimodal content propagation patterns hidden in user interaction behaviors, multimodal recommendation systems improve collaborative filtering performance by mining latent semantic information in multimedia features, revealing content propagation patterns and user interaction preferences based on content consumption records. Therefore, studying the relationship between multimodal content and user preferences remains a fundamental challenge for multimodal recommendation.

[0004] To bridge the gap between collaborative interaction modeling and multimedia information diffusion patterns, various multimodal recommendation methods have been proposed. Early attempts aimed to incorporate visual information into matrix factorization (e.g., VBPR [1] and ACF [2]). [2] ) in order to establish a connection between interaction records and multimodal content of items. With the popularity of Graph Neural Network (GNN), GNN-based multimodal recommendation has a significant advantage in exploring high-order interaction dependencies by combining multimodal features with user / item representations (user and item ID embedding vectors), becoming a mainstream solution. For example, MMGCN [3] and GRCN [4] The modality representation, diffusion, and aggregation paradigms were specified to refine the propagation paradigm. Subsequent studies (e.g., LATTICE [5] and FREEDOM [6] ) uses auxiliary item-item graphs to deeply explore the interdependencies between items. In order to further explore modal correlations, self-supervised learning is adopted as a data augmentation method to enhance the modality-aware recommendation performance.

[0005] Although existing multimodal recommendation models have achieved impressive performance by incorporating multimedia features into the representation embeddings of entities (users / items), these methods treat multimodal information as a fixed input source and mainly focus on propagating limited capacity multimodal information. As a result, they ignore the vast amount of knowledge contained in the multimodal information itself, leading to limited performance improvements. Specifically, most models typically rely on pre-trained visual models (e.g., ResNet [7] or ViT [8] ) and language models (e.g., BERT [9] ) to obtain visual and textual representations respectively. However, this reliance on access to multimodal representations may hinder the performance enhancement of recommendation models: (1) First, the semantic information of the multimodal itself is not fully utilized, but rather roughly extracted as a whole, resulting in insufficient utilization of contextual information, thereby limiting the potential of multimodal modeling. Taking the Amazon-Review dataset as an example, the text description contains various metadata, including category, brand, price, etc. The current typical multimodal recommendation processing method ignores the fine-grained feature details, directly inputs the data into the pre-trained language model, and uses the generated embedding as the feature vector of the multimodal recommendation model. This modality information modeling ignores fine-grained domain features and may introduce noise signals in the downstream training process. (2) Second, the semantic gap between multimodal features leads to fragmentation and semantic separation between individual unimodal features. Specifically, existing multimodal recommendation methods often process each unimodal representation separately without considering the semantic gap between different modal representations. This unimodal processing without cross-modal joint analysis cannot fully tap the large amount of available knowledge, thereby limiting performance improvement. Although many multimodal recommendation models strive to address the problem of inconsistent representation, the inconsistent input of multimodal representations directly hinders the performance of downstream recommendation models. Summary of the invention

[0006] In view of the above problems, the present invention proposes a multimodal recommendation method and system based on multi-perspective cross-modal semantic alignment, in an effort to solve or at least alleviate at least one of the above problems.

[0007] According to one aspect of the present invention, a multimodal recommendation method based on multi-view cross-modal semantic alignment is proposed, and the method comprises the following steps:

[0008] Extracting user information, item information, user-item interaction information, and multimodal auxiliary information of the training data, wherein the multimodal auxiliary information includes a textual modality and a visual modality;

[0009] Processing the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation;

[0010] Inputting user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training;

[0011] Use the trained multimodal neural network model for multimodal recommendation.

[0012] Furthermore, the processing of the multimodal auxiliary information includes:

[0013] Segmenting the text modality into multiple channels to obtain multiple single-channel text representations;

[0014] Encode the multiple single-channel text representations respectively to obtain multiple single-channel text encodings;

[0015] Encode the visual representation corresponding to the visual modality to obtain an encoded visual representation vector;

[0016] Calculate the similarity between each single-channel text encoding and visual representation vector;

[0017] Integrate multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector;

[0018] The integrated text encoding vector and the encoded visual representation vector constitute semantic information.

[0019] Furthermore, the encoding of single-channel text representation and visual representation is completed using a contrastive language image pre-training model.

[0020] Furthermore, the similarity between each single-channel text encoding and visual representation vector is calculated according to the following formula:

[0021]

[0022] in, represents the single-channel text encoding of the j-th channel of the i-th item, and t represents the text modality; represents the encoded visual representation vector; τ represents the temperature coefficient, which is used to balance the size of the vector representation; C represents the total number of channels; sim(·,·) represents the similarity calculation.

[0023] Furthermore, the similarity calculation is completed using a cosine similarity function.

[0024] Furthermore, based on the similarity, multiple single-channel text encodings are integrated using a self-attention operation:

[0025]

[0026] in, Represents the integrated text encoding vector.

[0027] According to another aspect of the present invention, a multimodal recommendation system based on multi-perspective cross-modal semantic alignment is proposed, the system comprising:

[0028] an information extraction module configured to extract user information, item information, user-item interaction information, and multimodal auxiliary information of the training data, wherein the multimodal auxiliary information includes a textual modality and a visual modality;

[0029] a multimodal auxiliary information processing module configured to process the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation;

[0030] A model training module, configured to input user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training;

[0031] The recommendation module is configured to perform multimodal recommendation using the trained multimodal neural network model.

[0032] Furthermore, the multimodal auxiliary information processing module processes the multimodal auxiliary information, including:

[0033] Segmenting the text modality into multiple channels to obtain multiple single-channel text representations;

[0034] Encode the multiple single-channel text representations respectively to obtain multiple single-channel text encodings;

[0035] Encode the visual representation corresponding to the visual modality to obtain an encoded visual representation vector;

[0036] Calculate the similarity between each single-channel text encoding and visual representation vector;

[0037] Integrate multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector;

[0038] The integrated text encoding vector and the encoded visual representation vector constitute semantic information.

[0039] Furthermore, the multimodal auxiliary information processing module uses a pre-trained model based on contrastive language images to complete the encoding of single-channel text representation and visual representation.

[0040] Furthermore, the multimodal auxiliary information processing module calculates the similarity between each single-channel text encoding and the visual representation vector according to the following formula:

[0041]

[0042] in, represents the single-channel text encoding of the j-th channel of the i-th item, and t represents the text modality; represents the encoded visual representation vector; τ represents the temperature coefficient, which is used to balance the size of the vector representation; C represents the total number of channels; sim(·,·) represents the similarity calculation;

[0043] Based on the similarity, multiple single-channel text encodings are integrated using self-attention operation: in, Represents the integrated text encoding vector.

[0044] The beneficial technical effects of the present invention are:

[0045] The present invention proposes a multimodal recommendation method and system based on multi-view cross-modal semantic alignment, and proposes a multimodal recommendation framework CLIPER enhanced by contrastive language image pre-training (CLIP). It establishes a connection between cross-modal representations from multiple viewpoints, and organically combines modality-aware representations with semantic similarity coefficients in multimodal recommendation. CLIP is incorporated into the multimodal recommendation problem and its cross-modal alignment capability is used to solve the semantic gap problem between modalities. A fine-grained modality semantic alignment method through multi-view semantic modeling is proposed; in addition, the representation of each specific viewpoint is combined with the similarity measurement coefficient; the CLIPER enhanced framework is seamlessly integrated with the existing multimodal recommendation model in a plug-and-play manner. Experimental results demonstrate the superiority and effectiveness of the CLIPER framework proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0047] Figure 1 The present invention is a flowchart of a multimodal recommendation method based on multi-view cross-modal semantic alignment according to an embodiment of the present invention.

[0048] Figure 2 This is an example diagram of multi-view metadata in an embodiment of the present invention.

[0049] Figure 3 It is the overall framework diagram of the CLIPER model in the embodiment of the present invention.

[0050] Figure 4 2 is a comparison chart of the semantic perspective ablation experiment results in an embodiment of the present invention.

[0051] Figure 5 This is a comparison chart of the impact of the embedding representation size on the model performance in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0053] Based on Contrastive Language-Image Pretraining (CLIP)

[10] Model, the present invention proposes a multimodal recommendation method and system based on multi-perspective cross-modal semantic alignment to complete the task of deep semantic information mining and the task of eliminating the semantic gap between different modes. The present invention deeply studies the extensive potential semantics in multimodal user-item interactions and simultaneously aligns different modal representations to better model the interaction between modalities. CLIP is a visual language joint pre-training model that is trained on 400 million image-text pairs and has semantic perception and cross-modal understanding capabilities.

[0054] In order to bridge the cross-modal semantic gap, the present invention proposes a CLIP-enhanced Recommender, abbreviated as CLIPER. Specifically, the present invention first targets the text modality, performs fine-grained modeling on the text description, segments the text features in units of fields, and regards each field of the text description as an observation perspective, so as to have a comprehensive understanding of the object representation from multiple perspectives; then, using the multi-perspective text description as a prompt and pairing it with the corresponding image as the input of CLIP, the pre-trained visual and language representation and the multi-perspective similarity metric embedding vector are obtained; finally, the different perspective representations are integrated through the fusion layer, and a plug-and-play recommendation enhancement framework is proposed, which is perfectly compatible with various backbone models of multi-modal recommendation.

[0055] The embodiment of the present invention proposes a multimodal recommendation method based on multi-view cross-modal semantic alignment, such as Figure 1 As shown, the method includes:

[0056] S1. extracting user information, item information, user-item interaction information and multimodal auxiliary information of training data, wherein the multimodal auxiliary information includes textual modality and visual modality;

[0057] S2. Processing the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation;

[0058] S3, inputting the user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training;

[0059] S4. Use the trained multimodal neural network model to perform multimodal recommendation.

[0060] The method starts from S1. In S1, user information, item information, user-item interaction information and multimodal auxiliary information of training data are extracted, and the multimodal auxiliary information includes textual modality and visual modality.

[0061] According to an embodiment of the present invention, first, given a user set Item Collection and the user-item interaction matrix R∈{0,1} M×N , each entry r in R ui =1 means user and items The positive interaction between and denote the number of users and items respectively. The goal of collaborative filtering without multimodal auxiliary information is to use the prediction score to predict the user's preference for uninteracted items. Similarly, multimodal recommendation aims to solve similar problems, but includes additional multimodal auxiliary information. To enhance item representation across different modalities. Here, Represents a modal collection A certain mode in the, the corresponding mode representation vector dimension is d m The present invention mainly considers text and visual modalities, i.e. t indicates text mode and v indicates visual mode.

[0062] The goal of multimodal recommendation is to predict missing user-item interaction records by leveraging multimodal features, expressed as This process is formalized as follows: The prediction function f(·) is used to model the multimodal recommendation task.

[0063] Then S2 is executed, in which the multimodal auxiliary information is processed to obtain semantic information including multimodal embedded text representation and visual representation. The processing of multimodal auxiliary information includes: segmenting the text modality into multiple channels to obtain multiple single-channel text representations; encoding the multiple single-channel text representations respectively to obtain multiple single-channel text encodings; encoding the visual representation corresponding to the visual modality to obtain an encoded visual representation vector; calculating the similarity between each single-channel text encoding and the visual representation vector; integrating the multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector; the integrated text encoding vector and the encoded visual representation vector constitute semantic information.

[0064] According to an embodiment of the present invention, in order to extract comprehensive semantic information at a fine-grained level, multi-view prompts are used for semantic exploration. The item metadata corpus is segmented by field and divided into C channels, such as Figure 2 As shown, each channel represents a single view. By pairing each individual cue with the corresponding image, we obtain a text-visual representation tuple Where j∈{0,1,…,C-1}. represents the textual hint of item i at view angle j (i.e., single-channel text representation), V i represents the cover image of item i. C represents the number of views (or channels), and each view is extracted by a separate prompt template. The overall model framework is as follows Figure 3 shown.

[0065] The text representation of any item i And visual representation Through the text encoder E t (·) and picture encoder E v (·) is obtained, and the specific method is as follows:

[0066]

[0067] Specifically, the text encoder E t (·) Responsible for converting multi-view text Encode and obtain the text encoding rich in semantic knowledge of item i from the perspective of j (i.e. single-channel text encoding). Similarly, the image encoder E v (·) Encode the cover image of item i to obtain a visual representation vector containing complex semantics Obtain text embeddings aligned in latent semantic space through the contrastive encoding module in CLIP And visual embedding Wherein, D=768 represents the dimension of the vector in CLIP.

[0068] It is worth noting that, unlike traditional studies that encode unimodal representations separately, the representations of a specific modality encoded by each encoder are aligned in a shared latent semantic space, sharing semantic knowledge. In addition, since a large number of image-text pairs are included in the pre-training stage of CLIP, these representations contain rich semantic information. On the one hand, the visual representation contains more complex details and less focused original information, thus providing a more comprehensive description. On the other hand, the textual representation is more precise and focused, but lacks sufficient content to describe all aspects of the image. Therefore, by splitting the textual description into multiple perspectives, it is possible to focus on different semantic focuses. The integration of various perspectives on the same item produces a more comprehensive representation.

[0069] Through contrastive learning, text encoding and image encoding are distributed in the same latent semantic space, so the representation vectors of different modalities gain semantic awareness that is lacking in downstream recommendation tasks. This is achieved by aligning the distribution of latent embeddings in the semantic space for the contrastive module of the text and image encoders. The similarity metric is used to evaluate the consistency of the semantic expression between the two embeddings:

[0070]

[0071] Among them, sim(·,·): Complete visual representation and text representation The similarity calculation of , the embodiment of the present invention uses the cosine similarity function cos(·,·) to achieve, that is, τ is the temperature coefficient used to balance the size of the vector representation during the similarity calculation.

[0072] By integrating the similarity scores between textual hints and object images at all viewpoints j∈{0,1,…,C-1} Get the similarity vector s i :

[0073]

[0074] The visual-text encoding fusion module integrates the various representations of the above encodings to form a comprehensive representation for downstream recommendation tasks, including text representation Visual Representation and semantically similar representation s i .

[0075] The visualization representation in the downstream recommendation backbone model is directly used To utilize its rich semantic information from CLIP. However, when it comes to text representation integration, the text description of a single perspective is often more one-sided and may not fully capture all semantics. Therefore, the self-attention (SA) operation is used for text representation integration as follows:

[0076]

[0077] For each multimodal encoding embedding, the corresponding attention coefficient is strengthened, and the weighted sum is output as the final text encoding vector

[0078] It should be noted that other methods can also be used for text representation integration, such as: sum (SUM) pooling operation, vector connection (Concat) operation, multi-layer perceptron (Multi-Layer, Perceptron, MLP) operation, etc. For SUM, Concat and MLP operations, since multimodal embedding vectors share the same semantic space and have the same vector dimension, the similarity representation is abandoned to maximize the retention of the original semantics, || represents the vector connection operator, and the output dimension is D; for the MLP operation, a single-layer MLP can be used, and LeakyReLU (·) is used as the activation function.

[0079] Then, S3 is executed, in which the user information, the item information, the user-item interaction information and the semantic information are input into a multimodal neural network model for training.

[0080] According to an embodiment of the present invention, the refined multimodal latent embedding i (include and ) as semantic information, and use these pre-trained embeddings as input to connect to the downstream recommendation backbone model, which can semantically represent the multimodal attributes of items. and As semantic information, it is then fed to the downstream recommendation backbone network. The representation vector extracted from the CLIPER model is seamlessly integrated into subsequent downstream tasks in a plug-and-play manner. The plug-and-play design ensures compatibility with any downstream multimodal recommendation model. The multimodal neural network model, i.e. the downstream recommendation backbone network, can choose mainstream models such as MMGCN, DualGNN, LATTICE, SLMRec, etc.

[0081] Finally, in S4, the trained multimodal neural network model is used for multimodal recommendation.

[0082] In summary, to emphasize the importance of eliminating the cross-modal semantic gap, this paper uses pre-trained cross-modal knowledge to enhance entity representation in the recommendation system, and introduces CLIP into the multi-modal recommendation scenario for the first time. In order to complete the integration of multi-modal recommendation semantic knowledge extraction and recommendation tasks, this paper proposes the CLIPER framework, which uses multi-view semantic representation connections to complete the alignment and enhancement of semantic representation, and completes the semantic unified representation framework under multi-modal recommendation.

[0083] The technical effect of the present invention is further verified through experiments.

[0084] The attribute corpus of the item is split into "fields" to obtain a fine-grained description of the item. Subsequently, unnecessary and noisy fields are filtered out, and only fields with rich semantics are retained for further processing. By treating each field as a single perspective, the value of each perspective is filled into the corresponding prompt template. Each field is treated as a separate "perspective" and its corresponding prompt template is filled with the corresponding value. Some specific prompt templates are provided below:

[0085] ·The product brand is_______.

[0086] ·The product categories are_______.

[0087] ·The product title is_______.

[0088] ·The product description is_______.

[0089] It is worth noting that although fields such as title and brand are indispensable in real-world e-commerce websites, in extreme cases, some low-quality recommendation datasets mix all fields together indiscriminately. In this case, in addition to tedious data preprocessing, it is recommended to use a global perspective prompt, i.e., "The product descriptions are_______.". The global perspective is also included in the subsequent experiments by concatenating all the individual perspectives mentioned previously. In addition, due to the character length limitation, the prompt is truncated to a maximum of 77 tokens.

[0090] To evaluate the performance of the CLIPER of the present invention, the effectiveness of the CLIPER model is verified on three real data sets, namely Amazon-Baby, Amazon-Sports and Amazon-Clothing. The data preprocessing and segmentation process is carried out according to the backbone model.

[0091] The widely used metrics Recall@K (R@K for short) and NDCG@K (N@K for short) are used to evaluate the recommendation performance, where K = 10, 20, 50 is used to ensure comprehensive comparison of different evaluation lengths. In the specific process, users with observable interactions are regarded as positive targets, and the remaining non-interaction behaviors are regarded as negative targets. After sorting the predicted interaction scores between users and all candidate items in descending order and masking the positive interaction scores observed in the training set, the predicted scores of the test set are calculated.

[0092] In order to verify the effectiveness of the CLIPER model, a total of 5 baseline methods were used for comparison. These 5 baseline methods are not only used as comparison targets, but also as the backbone network recommended by downstream. The performance of the model enhanced by CLIPER is compared with the original model to verify the effectiveness. Specifically, the 5 baseline methods are listed as follows:

[0093] MMGCN [3] : MMGCN is a classic graph-based multimodal recommendation method, which introduces a modality-specific graph to effectively propagate and aggregate each unimodal representation.

[0094] DualGNN

[11] : DualGNN introduces a user-user co-occurrence graph as an auxiliary structure to supplement the user-item bipartite graph, which can capture the correlation between users and facilitate the modeling of modal fusion patterns.

[0095] LATTICE [5] : LATTICE is widely regarded as a state-of-the-art multimodal recommendation method, which explicitly constructs a latent semantic item-item dynamic graph and effectively propagates modality-aware representations across the user-item interaction graph and item-item semantic graph.

[0096] SLMRec

[12] : SLMRec proposes a novel self-supervised learning-based data augmentation method that exploits multimodal distribution patterns to generate multiple perspectives, promotes contrastive representation modeling, and enables a comprehensive understanding of the potential interrelationships between different modalities.

[0097] FREEDOM [6] :Based on LATTICE, FREEDOM proposes to freeze the item-item semantic graph and denoise the user-item interaction graph, thereby enhancing the modeling of modal-aware representation propagation.

[0098] Since the maximum length in CLIP is limited to 77 tokens in length, Long-CLIP is selected

[13] As the default backend model, it allows an input length of up to 248 tokens. Two versions of the image encoder in CLIP are provided, ResNet and ViT, and the ViT-L / 14 and Sentence Transformer models are selected as the visual and text encoders of CLIP, respectively. In addition, the LongCLIP-L version is used for the LongCLIP experimental implementation. Regarding the design of the visual-text encoding fusion module, all four integration methods were explored, but the SA operation was mainly adopted. The temperature coefficient τ was fine-tuned in the range of {0.05, 0.1, 0.2, 0.5, 0.7} to find the temperature coefficient corresponding to the best performance; in addition, in order to ensure the fairness of the comparison, the embedding dimension d of the downstream recommendation model was fixed in the range of {128, 256, 512, 1024} and fine-tuned to ensure performance.

[0099] Based on the official code or MMRec for all backbone models

[14] The framework implements and fine-tunes all baseline models. The performance of all baseline models has been fine-tuned. The performance of CLIPER-enhanced baseline models is compared with their original counterparts. Specifically, for the tuning of each backbone model, the Adam optimizer is used. The early-stop strategy is used for evaluation. If there is no improvement in the indicator Recall@20 after 10 consecutive epochs, the training stops. The learning rate lr is searched in the range of {1,3,5}×{1e-4,1e-5}. By default, the weight decay coefficient λ of the L2 regularization term is set to 1e-4.

[0100] Tables 1, 2, and 3 show the performance comparison of CLIPER based on different backbone models on different datasets.

[0101] Table 1. Comparison of overall performance of models on the Amazon-Baby dataset

[0102]

[0103] Table 2. Comparison of overall performance of models on the Amazon-Sports dataset

[0104]

[0105] Table 3. Comparison of overall performance of models on the Amazon-Clothing dataset

[0106]

[0107]

[0108] Comparative analysis of performance leads to the following conclusions:

[0109] 1) CLIPER shows consistent performance improvement in multimodal recommendation scenarios based on multiple backbone models. On Amazon-Baby, Recall@K and NDCG@K are improved by 8.40% and 10.50% on average compared with the backbone models not enhanced by CLIPER, respectively; on Amazon-Sports, Recall@K and NDCG@K have an average improvement of 7.17% and 8.32% on average; on Amazon-Clothing, Recall@K and NDCG@K have an average improvement of 8.48% and 7.99% on average compared with all baseline models. The performance improvement achieved by FREEDOM-CLIPER is particularly noteworthy, achieving a significant improvement of 35.33% on Recall@50 over the original FREEDOM model. The experiments have strongly verified the effectiveness of the CLIPER model proposed in this paper.

[0110] 2) CLIPER enhances the backbone models on different datasets, demonstrating its ability to unleash the modeling potential of various backbone models. Specifically, CLIPER-enhanced DualGNN, LATTICE, and FREEDOM achieve huge performance improvements on the Baby, Sports, and Clothing datasets, respectively. On the Baby dataset, DualGNN-CLIP ER achieves an average improvement of 16.26% and 19.82% over the DualGNN model in Recall@K and NDCG@K. Similarly, on the Sports dataset, LATTICE-CLIPER achieves an improvement of 10.94% and 13.83% over LATTICE in Recall@K and NDCG@K, respectively. In addition, on the Clothing dataset, FREEDOM-CLIPER improves Recall@K and NDCG@K by 18.36% and 15.59% over the FREEDOM model, respectively. The consistent enhancement of different baseline models highlights the versatility of CLIPER on a wide range of data.

[0111] 3) FREEDOM-CLIPER shows the best performance. The best performing baseline model FREEDOM shows the highest performance in all scenarios, where both the original version and the CLIPER-enhanced version of FREEDOM outperform other multimodal recommendation methods. The CLIPER framework further expands its superiority over other multimodal recommendation methods. It is worth noting that on Clothing, CLIPER achieves significant improvements of 35.33% and 29.50% on Recall@50 and NDCG@50, respectively. This shows that CLIPER has a strong semantic understanding ability for longer target sequences.

[0112] 4) CLIPER brings the most significant performance improvement on the Baby dataset. Specifically, on Baby, CLIPER achieves an average improvement of 8.40% and 10.50% on Recall@K and NDCG@K, respectively.

[0113] The impact of various key architectures of the CLIPER model is further verified, including the analysis of semantic perspective and the analysis of visual-text encoding fusion design. Given that FREEDOM-CLIPER exhibits the highest multimodal recommendation performance, FREEDOM-CLIPER is used as the default CLIPER model implementation for subsequent model architecture and parameter sensitivity analysis.

[0114] CLIPER's comprehensive semantic exploration of multimodal auxiliary information is achieved by extracting different semantic perspectives. In order to study the impact of perspective on the results, an ablation experiment was conducted on the Amazon-Baby dataset. By removing a specific perspective [v], the corresponding variant model is denoted as CLIPER-[v]. The experimental perspectives include "title", "brand", "description", "categories", and an unrefined global view is added. In addition, in order to verify the effectiveness of multi-perspective semantic extraction, a CLIEPR variant with only a global perspective is explored and denoted as CLIPER-only-global. Figure 4 Performance comparison of ablation experiments on the semantic perspective is shown.

[0115] The consistent superiority of CLPER over all other variants confirms the effectiveness of fine-grained multi-view semantic modeling. In addition, the different magnitudes of performance degradation across different variants suggest that the priority of perspectives is as follows: title > brand > description > category. Titles contain more refined and condensed information, while categories are often composed of repetitive and low-value categorical information. In addition, the limited impact of removing the global perspective further validates the effectiveness of fine-grained multi-view processing. In addition, the performance of CLIPER-only-global is consistently lower than that of other model variants, which also proves that coarse-grained semantic extraction may hinder the improvement of recommendation performance.

[0116] Further analysis experiments are conducted on the key parameters of CLIPER, including the embedding size d and the temperature coefficient τ, and the optimal parameter settings are determined through comparison on the Clothing dataset.

[0117] First, the model parameters are tuned in the range of {128, 256, 512, 1024} for the optimal value of the embedding size d. Figure 5As shown in the figure, within the range of d<512, as d increases, CLIPER performance continues to improve; when d>512, the model tends to deteriorate rapidly. Setting the dimension value d=512 brings the best performance. Analyzing this phenomenon, increasing the embedding size can enhance the ability of the recommendation model. However, too large a value may cause overfitting problems and affect the reasoning ability of the model.

[0118] Secondly, the similarity measurement score can balance the calculation of multiple view similarities by combining the temperature coefficient τ. The optimal value of τ is debugged in the range of 0.05, 0.1, 0.2, 0.5, and 0.7, and a comparative analysis is given in Table 4. In most cases, setting τ = 0.1 can produce the best performance; however, in some cases, setting τ = 0.05 corresponds to excellent performance. When τ>0.1, CLIPER performance tends to deteriorate. In the present invention, τ = 0.1 is uniformly set for similarity measurement.

[0119] Table 4. Comparison of the impact of temperature coefficient τ on model performance

[0120]

[0121] Another embodiment of the present invention provides a multimodal recommendation system based on multi-perspective cross-modal semantic alignment, the system comprising:

[0122] an information extraction module configured to extract user information, item information, user-item interaction information, and multimodal auxiliary information of the training data, wherein the multimodal auxiliary information includes a textual modality and a visual modality;

[0123] a multimodal auxiliary information processing module configured to process the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation;

[0124] A model training module, configured to input user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training;

[0125] The recommendation module is configured to perform multimodal recommendation using the trained multimodal neural network model.

[0126] In this embodiment, preferably, the multimodal auxiliary information processing module processes the multimodal auxiliary information including:

[0127] Segmenting the text modality into multiple channels to obtain multiple single-channel text representations;

[0128] Encode the multiple single-channel text representations respectively to obtain multiple single-channel text encodings;

[0129] Encode the visual representation corresponding to the visual modality to obtain an encoded visual representation vector;

[0130] Calculate the similarity between each single-channel text encoding and visual representation vector;

[0131] Integrate multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector;

[0132] The integrated text encoding vector and the encoded visual representation vector constitute semantic information.

[0133] In this embodiment, preferably, the multimodal auxiliary information processing module uses a pre-trained model based on contrastive language images to complete the encoding of single-channel text representation and visual representation.

[0134] In this embodiment, preferably, the multimodal auxiliary information processing module calculates the similarity between each single-channel text encoding and the visual representation vector according to the following formula:

[0135]

[0136] in, represents the single-channel text encoding of the j-th channel of the i-th item, and t represents the text modality; represents the encoded visual representation vector; τ represents the temperature coefficient, which is used to balance the size of the vector representation; C represents the total number of channels; sim(·,·) represents the similarity calculation;

[0137] Based on the similarity, multiple single-channel text encodings are integrated using self-attention operation: in, Represents the integrated text encoding vector.

[0138] The function of a multimodal recommendation system based on multi-perspective cross-modal semantic alignment described in this embodiment can be explained by the aforementioned multimodal recommendation method based on multi-perspective cross-modal semantic alignment. Therefore, for the parts not described in detail in this embodiment, please refer to the above method embodiments and will not be repeated here.

[0139] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

[0140] The documents cited in the present invention are as follows:

[0141] [1]He R,McAuley J J.VBPR:Visual Bayesian Personalized Ranking fromImplicit Feedback[C] / / Proceedings ofthe 30thAAAI Conference onArtificialIntelligence.2016:144-150.[2]Chen J,Zhang H,He X,et al.AttentiveCollaborative Filtering:Multimedia Recommendation with Item-and Component-Level Attention[C] / / Proceedings of the 40th International ACM SIGIRConference on Research and Development in Information Retrieval.2017:335-344.

[0142] [3]Wei Y,Wang X,Nie L,et al.MMGCN:Multi-modal Graph ConvolutionNetwork for Personalized Recommendation of Micro-video[C] / / Proceedings of the27th ACM International Conference on Multimedia.2019:1437-1445.

[0143] [4]Wei Y,Wang X,Nie L,et al.Graph-Refined Convolutional Network forMultimedia Recommendation with Implicit Feedback[C] / / Proceedings of the 28thACM International Conference on Multimedia.2020:3541-3549.

[0144] [5]Zhang J,Zhu Y,Liu Q,et al.Mining Latent Structures for MultimediaRecommendation[C] / / Proceedings ofthe 29thACM International Conference onMultimedia.2021:3872-3880.

[0145] [6]Zhou X,Shen Z.A Tale of Two Graphs:Freezing and Denoising GraphStructures for Multimodal Recommendation[C] / / Proceedings of the 31st ACMInternational Conference on Multimedia.2023:935-943.

[0146] [7]He K,Zhang X,Ren S,et al.Deep Residual Learning for ImageRecognition[C] / / Proceedings ofthe 2016 IEEE Conference on Computer Vision andPattern Recognition.2016:770-778.

[0147] [8]Dosovitskiy A,Beyer L,Kolesnikov A,et al.An Image is Worth 16x16Words:Transformers for Image Recognition at Scale[C] / / Proceedings of the 9thInternational Conference on Learning Representations.2021.

[0148] [9]Devlin J,Chang M W,Lee K,et al.BERT:Pre-training ofDeepBidirectional Transformers for Language Understanding[C] / / Proceedings of the2019 Conference of the North American Chapter of the Association forComputational Linguistics:Human Language Technologies.2019:4171-4186.

[0149]

[10] Radford A,Kim J W,Hallacy C,et al.Learning Transferable VisualModels From Natural Language Supervision[C] / / Proceedings ofthe 38thInternational Conference on Machine Learning:Vol.139.2021:8748-8763.

[0150]

[11] Wang Q,Wei Y,Yin J,et al.DualGNN:Dual Graph Neural Network forMultimedia Recommendation[J].IEEE Transactions on Multimedia,2021,25:1074-1084.

[0151]

[12] Tao Z,Liu X,Xia Y,et al.Self-Supervised Learning for MultimediaRecommendation[J].IEEE Transactions on Multimedia,2023,25:5107-5116.

[0152]

[13] Zhang B,Zhang P,Dong X,et al.Long-CLIP:Unlocking the Long-TextCapability ofCLIP[J].arXiv preprint arXiv:2403.15378,2024.

[0153]

[14] Zhou H,Zhou X,Zeng Z,et al.A Comprehensive Survey on MultimodalRecommender Systems:Taxonomy,Evaluation,and Future Directions[J].CoRR,2023,abs / 2302.04473.

Claims

1. A multimodal recommendation method based on multi-view cross-modal semantic alignment, characterized in that: The following steps are involved: Extracting user information, item information, user-item interaction information, and multimodal auxiliary information of the training data, wherein the multimodal auxiliary information includes a textual modality and a visual modality; Processing the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation; The processing of the multimodal auxiliary information includes: segmenting the text modality into multiple channels to obtain multiple single-channel text representations; encoding the multiple single-channel text representations respectively to obtain multiple single-channel text encodings; encoding the visual representation corresponding to the visual modality to obtain an encoded visual representation vector; calculating the similarity between each single-channel text encoding and the visual representation vector; integrating the multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector; the integrated text encoding vector and the encoded visual representation vector constitute semantic information; Inputting user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training; Use the trained multimodal neural network model for multimodal recommendation.

2. According to claim 1, a multimodal recommendation method based on multi-view cross-modal semantic alignment is characterized in that: The encoding of single-channel textual representation and visual representation is completed using a contrastive language-image pre-trained model.

3. According to claim 1, a multimodal recommendation method based on multi-view cross-modal semantic alignment is characterized in that: The similarity between each single-channel text encoding and visual representation vector is calculated according to the following formula: in, represents the single-channel text encoding of the j-th channel of the i-th item, and t represents the text modality; represents the encoded visual representation vector; τ represents the temperature coefficient, which is used to balance the size of the vector representation; C represents the total number of channels; sim(·,·) represents the similarity calculation.

4. According to claim 3, a multimodal recommendation method based on multi-view cross-modal semantic alignment is characterized in that: The similarity calculation is performed using a cosine similarity function.

5. According to claim 3, a multimodal recommendation method based on multi-view cross-modal semantic alignment is characterized in that: Based on the similarity, multiple single-channel text encodings are integrated using self-attention operation: in, Represents the integrated text encoding vector.

6. A multimodal recommendation system based on multi-view cross-modal semantic alignment, characterized in that: include: an information extraction module configured to extract user information, item information, user-item interaction information, and multimodal auxiliary information of the training data, wherein the multimodal auxiliary information includes a textual modality and a visual modality; a multimodal auxiliary information processing module configured to process the multimodal auxiliary information to obtain semantic information including multimodal embedded text representation and visual representation; The processing of the multimodal auxiliary information includes: Segmenting the text modality into multiple channels to obtain multiple single-channel text representations; Encode the multiple single-channel text representations respectively to obtain multiple single-channel text encodings; Encode the visual representation corresponding to the visual modality to obtain an encoded visual representation vector; Calculate the similarity between each single-channel text encoding and visual representation vector; Integrate multiple single-channel text encodings based on the similarity to obtain an integrated text encoding vector; The integrated text encoding vector and the encoded visual representation vector constitute semantic information; A model training module, configured to input user information, item information, user-item interaction information and the semantic information into a multimodal neural network model for training; The recommendation module is configured to perform multimodal recommendation using the trained multimodal neural network model.

7. The multimodal recommendation system based on multi-view cross-modal semantic alignment according to claim 6, characterized in that: The multimodal auxiliary information processing module uses a pre-trained model based on contrastive language images to complete the encoding of single-channel text representation and visual representation.

8. The multimodal recommendation system based on multi-view cross-modal semantic alignment according to claim 7, characterized in that: The multimodal auxiliary information processing module calculates the similarity between each single-channel text encoding and visual representation vector according to the following formula: in, represents the single-channel text encoding of the j-th channel of the i-th item, and t represents the text modality; represents the encoded visual representation vector; τ represents the temperature coefficient, which is used to balance the size of the vector representation; C represents the total number of channels; sim(·,·) represents the similarity calculation; Based on the similarity, multiple single-channel text encodings are integrated using self-attention operation: in, Represents the integrated text encoding vector.

Images