Multi-modal alignment service recommendation method based on prompt learning

Abstract

The invention discloses a multi-modal alignment service recommendation method based on prompt learning, which comprises the following steps of: firstly, acquiring text and image data of articles historically clicked by a user, and respectively obtaining text features and image features in combination with prompt word vectors; secondly, inputting the text features and the image features into two encoders, respectively outputting multi-modal feature representations of the text and the image, carrying out feature enhancement through discarding operation, and respectively inputting the features into a text optimizer and an image optimizer; and then similarity calculation is carried out on the features obtained by the two optimizers, a contrast loss function is calculated, the features output by the optimizers are spliced and fused to generate article representation, and a sequence representation of a user is obtained through a dynamic clustering algorithm in combination with a Transform-based auto-encoder. And finally, designing two self-supervised contrast learning tasks to carry out loss calculation and training. According to the method, interests and demands of the user are more accurately captured, and more personalized recommendation is provided.

Description

Technical Field

[0001] The present invention belongs to the field of network services and relates to a multimodal alignment service recommendation method based on prompt learning, which aims to achieve dynamic and personalized service recommendation by optimizing multimodal data. Background Art

[0002] In recent years, multimodal recommendation systems have become increasingly prominent in the fields of artificial intelligence and information retrieval, becoming a hot research topic. By integrating heterogeneous information from multiple sources, such as text and images, these systems can construct more comprehensive and in-depth descriptions of users and items. This ability to integrate multidimensional information gives multimodal recommendation systems a unique advantage in understanding user preferences and improving recommendation quality.

[0003] In real-world applications, text and image information are ubiquitous and play an indispensable role in attracting user attention and enhancing the user experience. An engaging title or description can significantly increase user engagement and interest, while visual elements such as image color, shape, and layout can play a decisive role in the user's decision-making process. With the rapid development of computer vision and natural language understanding technologies, research results in these fields have provided powerful technical support for extracting content semantics from text and images, laying the foundation for the development of multimodal recommendation systems.

[0004] However, despite the enormous potential of multimodal recommender systems, existing research and methods still face numerous challenges in aligning multimodal information and fusion. Multimodal synergy varies from user to user and is dynamic: the appeal of each item's multimodal information to users is user-dependent and dynamic. The same item may attract different users for different reasons, and each modality may contribute unequally to user interest. Even for the same user-item pair, this pattern can vary over time and context, making the design of effective multimodal fusion methods crucial. Information redundancy can also obscure users' core interests: interaction sequences often exhibit an unbalanced distribution of interests. User interaction sequences typically contain a large number of homogeneous items (such as daily necessities), while other, more informative items (such as sporting goods) may only appear occasionally. Treating all interactions equally in user behavior modeling can lead to overemphasis on certain categories of items and underattention to others. In multimodal sequential recommender systems, information representation and semantic spaces of different modalities often differ, making direct alignment difficult.

[0005] Early multimodal sequential recommendation systems typically employed simple feature concatenation or weighted summation methods to directly integrate information from each modality into the model, failing to effectively model the complex interactions between modalities. This simplistic fusion approach overlooks semantic alignment and context-dependent nature of the modalities. For example, visual features may play a primary role in recommendation results in some situations, while textual features are more important in others. Furthermore, multimodal data may contain redundant or conflicting information, raising the question of how to select the modal information most valuable for recommendation tasks while suppressing redundancy and noise. Furthermore, the design of multimodal interaction mechanisms must consider the diversity and dynamic nature of user interests. Early multimodal alignment methods failed to account for the differences in feature spaces between modalities. For example, visual information in an image may not directly correspond to the abstract concepts described in text. Alignment involves more than simple data matching; it requires a deep understanding of the semantic content across modalities, which necessitates complex algorithms and models. Summary of the Invention

[0006] To address the above issues, this paper combines hint learning with multimodal sequence recommendation. Hint learning involves adding hints to language or visual language models to better align multimodal data. Hints guide the visual and language branches to move closer together in the feature space. Hints are then optimized through end-to-end training, enabling the model to better adapt to downstream tasks.

[0007] The goal is to use cue learning to bring the representations of multimodally consistent samples closer together in the representation space, while dissimilar samples become more distant, thereby achieving better extraction of model representations. Previous methods have mostly performed cue learning independently in either the language or vision branch, failing to fully exploit the multimodal nature of the vision-text model. This invention achieves better adaptation to downstream tasks by simultaneously performing cue learning in both the language and vision branches.

[0008] The purpose of this invention is to provide users with more personalized and efficient service recommendations by comprehensively analyzing and utilizing data from multiple modalities, and to propose a multimodal sequence recommendation system based on modality alignment.

[0009] The use steps of the method of the present invention are:

[0010] Step 1: Obtain the text and image data of the user's historical purchases, and combine them with the prompt word vector to obtain text features and image features respectively.

[0011] The text and image data of a user's purchased items are arranged in chronological order. The text information is processed using the CLIP model and converted into a high-dimensional text feature representation. Similarly, the image data is converted into an image feature representation using the CLIP model, capturing the visual content of the item. Before model training begins, a set of random hint word vectors are initialized for the text and image data, respectively. Finally, the hint word vectors are combined with the corresponding text feature representation and image feature representation to form text and image features.

[0012] Step 2: Input the text features and image features into two Transformer encoders respectively, perform feature extraction, and output multimodal feature representations of text and image.

[0013] VL coupling function: After encoding each layer of the Transformer encoder, the text and image prompt word vectors are adjusted according to the output features of the current layer through the VL coupling function F to achieve cross-modal feature alignment: this function F is composed of multiple linear layers, and the text prompt word vector P is mapped to the visual feature space through a projection matrix, with the same dimension as the image prompt word vector, so that the prompt words of the two modalities share a unified embedding representation; the two Transformer encoders output the final adjusted multimodal feature representation of text and image.

[0014] Step 3: Apply the discard operation to the multimodal feature representation of text and image for feature enhancement, and then input them into the text optimizer and image optimizer respectively.

[0015] Dropout operations are applied twice to the feature vectors of text and image for feature enhancement, thereby generating two positive sample views.

[0016] The enhanced text and image features are fed into the optimizer.

[0017] Step 4: Calculate the similarity of the features obtained by the two optimizers, calculate the comparison loss function based on the similarity, and concatenate and fuse the features output by the two optimizers to generate the item representation.

[0018] Step 5: Item representation is mined through a dynamic clustering algorithm to identify interest representation sequences.

[0019] The unified item representation obtained in step 4 is used through a dynamic clustering algorithm to automatically mine users' implicit multimodal interest patterns. First, before each round of training, an improved density peak clustering algorithm (DPC-KNN) is used to perform cluster analysis on the item representations of all products. By calculating the local density and minimum distance of each product token, the top K points are selected as core interest prototypes. Then, during model training and inference, the item representations are mapped to these pre-calculated interest prototypes in real time, thereby converting lengthy product browsing records into compact interest representation sequences.

[0020] Step 6: Based on the Transformer encoder-decoder model, obtain the user's sequence representation.

[0021] We designed and implemented a Transformer-based encoder-decoder model for multimodal interest-aware sequence representation. We used interest representation sequences as keys and values, combined with item representations and location embeddings to generate decoding queries, and finally obtained the user's sequence representation through the output of the last query token.

[0022] Step 7: Based on the sequence representation, design two self-supervised contrastive learning tasks for loss calculation and training.

[0023] To learn universal representations of multimodal interaction sequences, two self-supervised contrastive learning tasks are designed to pre-train the model.

[0024] Sequence-item comparison learning: To capture the correspondence between interaction sequences and candidate items, sequence-item CL is used as a pre-training task.

[0025] Sequence-sequence contrastive learning: Leveraging semantic invariance as a self-supervisory signal to enhance the robustness of sequence models.

[0026] The beneficial effects of the present invention are: by utilizing prompt learning and training only the prompt word part, the text and image are effectively aligned at the feature level, so that the model can better understand and utilize multimodal information and use fewer parameters, thereby more accurately capturing the user's interests and needs and providing more personalized recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Multimodal recommendation method based on prompt word pre-training;

[0028] Figure 2 Tips for studying the schematic diagram. DETAILED DESCRIPTION

[0029] The following further describes a multimodal alignment service recommendation method based on prompt learning proposed by the present invention, which includes the following steps:

[0030] Step 1: Obtain the text and image data of the user's historical purchases, and combine them with the prompt word vector to obtain text features and image features respectively.

[0031] The text and image data of a user's purchased items are arranged in chronological order. The text information is processed using the CLIP model and converted into a high-dimensional text feature representation, preserving the rich semantics of the item description. Similarly, the image data is converted into an image feature representation using the CLIP model, capturing the item's visual content. Before model training begins, a set of random cue word vectors are initialized for the text and image data, respectively. Finally, the cue word vectors are combined with the corresponding text and image feature representations to form text and image features. These cue word vectors serve as the starting point for model learning and guide the alignment of multimodal features.

[0032] Step 2: Input the text features and image features into two Transformer encoders respectively, perform feature extraction, and output multimodal feature representations of text and image.

[0033] Cue word learning: In the Transformer encoding layer, each layer's input is a cue word and a text or image vector. The cue word input to the first layer is randomly initialized. With each subsequent layer, the text and image cue word vectors are adjusted based on the output features of the current layer. This process allows the cue word to gradually adapt to the key information extracted by the model from the text and image.

[0034] VL coupling function: After encoding each layer of the Transformer encoder, the text and image prompt word vectors are adjusted according to the output features of the current layer through the VL coupling function F to achieve cross-modal feature alignment: this function F is composed of multiple linear layers, and the text prompt word vector P is mapped to the visual feature space through a projection matrix, with the same dimension as the image prompt word vector, so that the prompt words of the two modalities share a unified embedding representation; the two Transformer encoders output the final adjusted multimodal feature representation of text and image.

[0035] After each encoding layer, the VL coupling function F is applied to adjust the visual cue word vector based on the adjusted text cue word vector. This step promotes synergy between text and image features, allowing for better alignment of features from both modalities. This continues through all encoding layers. As the number of layers increases, the cue word vectors are continuously adjusted based on the feature information extracted by the model, achieving more accurate multimodal feature alignment. After processing through the encoding layers, the final adjusted multimodal feature representations of text and image are output.

[0036] Step 3: Apply the discard operation to the multimodal feature representation of text and image for feature enhancement, and then input them into the text optimizer and image optimizer respectively.

[0037] Dropout operations are applied twice to the feature vectors of text and image for feature enhancement, thereby generating two positive sample views.

[0038] The enhanced text and image features are fed into the optimizer. The optimizer is a linear transformation layer that projects the high-dimensional text and image features obtained in step 2 into a unified latent representation space, thereby aligning and fusing features from different modalities in the same dimensional space. Its internal parameters perform nonlinear transformations on the features, extracting higher-level abstract features that are more suitable for subsequent fusion and recommendation tasks. Similarly, the enhanced image features are fed into the image optimizer.

[0039] Step 4: Calculate the similarity of the features obtained by the two optimizers, calculate the comparison loss function based on the similarity, and concatenate and fuse the features output by the two optimizers to generate the item representation.

[0040] To assess the degree of correspondence between image and text features, a similarity calculation is performed on the features obtained by the image and text optimizers. A contrastive loss function is used. The calculated loss value is then used during backpropagation to update the model parameters and optimize its performance.

[0041]

[0042] Temperature represents the temperature coefficient (hyperparameter), which is used to adjust the sharpness of the probability distribution and affect the learning intensity of difficult samples. Represents the similarity score of the i-th positive sample pair. It is calculated by normalizing the dot product of image and text features. A larger value indicates a higher match. It represents the similarity score of the j-th negative sample pair, which is the score of the mismatched image-text pair. The model needs to lower such scores.

[0043] The converted features output by the text optimizer and the image optimizer are concatenated and fused to generate a unified and comprehensive item representation.

[0044] Step 5: Item representation is mined through a dynamic clustering algorithm to identify interest representation sequences.

[0045] The unified item representations obtained in step 4 are used through a dynamic clustering algorithm to automatically mine users' implicit multimodal interest patterns. First, before each round of training, an improved density peak clustering algorithm (DPC-KNN) is used to cluster the item representations of all products. By calculating the local density and minimum distance of each product token, the top K points are selected as core interest prototypes. Then, during model training and inference, the item representations are mapped to these pre-computed interest prototypes in real time, transforming lengthy product browsing records into compact interest representation sequences. This design not only preserves the multimodal semantic information in user behavior, but also effectively filters noise through interest clustering, ultimately outputting a sequence of interest tokens with clear semantic interpretation and low redundancy for subsequent recommendation prediction.

[0046] Step 6: Based on the Transformer encoder-decoder model, obtain the user's sequence representation.

[0047] We designed and implemented a Transformer-based encoder-decoder model for multimodal interest-aware sequence representation. We used interest representation sequences as keys and values, combined with item representations and location embeddings to generate decoding queries, and finally obtained the user's sequence representation through the output of the last query token.

[0048] Step 7: Based on the sequence representation, design two self-supervised contrastive learning tasks for loss calculation and training.

[0049] To learn universal representations of multimodal interaction sequences, two self-supervised contrastive learning tasks are designed to pre-train the model.

[0050] Sequence-item comparison learning: To capture the correspondence between interaction sequences and candidate items, sequence-item comparison learning (CL) is used as a pre-training task:

[0051]

[0052] The sequence representation u of user i obtained in step 6 is represented by i Instead of actually buying items The similarity between the user's interests and the target items is measured, with larger values indicating a higher degree of match. τ is a hyperparameter that adjusts the distribution of similarity scores.

[0053] Sequence-sequence contrastive learning: Using semantic invariance as a self-supervisory signal to enhance the robustness of sequence models. Given u i and u′ i As the representation of the two enhanced sequences of the i-th user:

[0054]

[0055] i ,u′ i > The similarity between two augmented sequence representations of the same user. This is used to capture the semantic invariance of user behavior (e.g., stability after temporal perturbations or modal augmentation). Indicates the original sequence list of other users in the current batch and u i The sum of similarities.

[0056] The training objectives are summarized as follows:

[0057]

[0058] represents the contrast loss between the user sequence representation and the true next item, represents the contrastive loss of two enhanced sequence views of the same user, <θ m ,θ m′ > represents the mean cosine similarity between interests, l Multiprompt Contrastive loss for cross-modal cue word alignment.

[0059] The present invention will be further described below with reference to the accompanying drawings:

[0060] (1) The overall process of the present invention is as shown in the attached Figure 1 shown.

[0061] First, using the advanced pre-trained CLIP model, we extract high-dimensional feature vectors for both text and images, and randomly initialize prompt words. Next, through interactive learning of text prompt words and image prompt words, we continuously adjust the embedding between text and image features, ensuring precise alignment of information from both modalities in the feature space. We then cluster the fused features based on interest, and design and implement a Transformer-based encoder-decoder model for multimodal interest-aware sequence representation. Through repeated model iterations, the system continuously optimizes parameters, ultimately outputting sequential recommendation results that closely match user interests.

[0062] (2) Tips on learning methods, such as Figure 2 shown.

[0063] First, b learnable hint tokens P are introduced in the text encoding branch. i , and concatenate them with the fixed word embedding W0 of the text to form a new input embedding. Then, the prompt token is injected into each Transformer block L of the text encoder i In this way, the output of each block [P i ,W i ] are based on the output of the previous block [P​i-1 ,W i-1 ] is processed. After passing through J layers, subsequent Transformer layers will continue to process the output containing the prompt token, and finally obtain the final text representation z by projecting the last word embedding of the last Transformer block to the common VL latent embedding space. In order to promote the synergy between visual and language cues, the VL cue coupling technology is adopted. The language cues P are mapped to visual cues through the VL coupling function F, and these cues are iteratively processed in the first J Transformer blocks, so that the visual and language branches can promote each other in the shared embedding space, and finally obtain the image representation x. This process not only takes into account the independent context of text and image, but also realizes the interaction and collaborative learning between the two modalities through shared cues.

Claims

1. A multimodal alignment service recommendation method based on prompt learning, characterized in that: The following steps are involved: Step 1: Obtain the text and image data of the user's historical purchases, and combine them with the prompt word vector to obtain text features and image features respectively; Step 2: Input text features and image features into two Transformer encoders respectively, perform feature extraction, and output multimodal feature representations of text and image; Step 3: Apply a dropout operation to the multimodal feature representations of text and image to perform feature enhancement, and then input them into the text optimizer and image optimizer respectively; Step 4: Calculate the similarity of the features obtained by the two optimizers, calculate the comparison loss function based on the similarity, and concatenate and fuse the features output by the two optimizers to generate the item representation; Step 5: Item representation uses a dynamic clustering algorithm to mine interest representation sequences, and then uses the Transformer encoder-decoder model to obtain the user's sequence representation. Step 6: Based on the sequence representation, design two self-supervised contrastive learning tasks for loss calculation and training.

2. The multimodal alignment service recommendation method based on prompt learning according to claim 1 is characterized in that The specific implementation of step 1 is as follows: first, the text and image data of the user's historical clicked items are arranged in chronological order; the text data is processed through the CLIP model and converted into text feature representation; the image data is converted into image feature representation through the CLIP model; secondly, a set of random prompt word vectors are initialized for the text and image data respectively; finally, the prompt word vectors are combined with the corresponding text feature representation and image feature representation to form text features and image features.

3. The method for recommending multimodal alignment services based on prompt learning according to claim 2, characterized in that: In the Transformer encoder, after encoding each layer of the Transformer encoder, the text and image prompt word vectors are adjusted according to the output features of the current layer through the VL coupling function F to achieve cross-modal feature alignment: the function F is composed of multiple linear layers, and the text prompt word vector P is mapped to the visual feature space through a projection matrix, with the same dimension as the image prompt word vector, so that the prompt words of the two modalities share a unified embedding representation; the two Transformer encoders output the final adjusted multimodal feature representation of the text and image.

4. The method for recommending multimodal alignment services based on prompt learning according to claim 3, characterized in that: The text optimizer and image optimizer are both composed of linear transformation layers, which project the enhanced text features and image features into a unified hidden representation space to achieve alignment and fusion of different modal features in the same dimensional space.

5. The multimodal alignment service recommendation method based on prompt learning according to claim 4 is characterized in that: The automatic mining of interest representation sequences is specifically implemented as follows: the unified item representation obtained in step 4 is used to automatically mine the user's implicit multimodal interest patterns through a dynamic clustering algorithm. First, before each round of training, the improved density peak clustering algorithm DPC-KNN is used to perform cluster analysis on the item representations of all products. By calculating the local density and minimum distance of each product, the highest K points are selected as core interest prototypes. Then, during the training and inference process, the item representations are mapped to the pre-calculated interest prototypes in real time, and the product browsing records are converted into interest representation sequences.

6. The method for recommending multimodal alignment services based on prompt learning according to claim 5, characterized in that: The specific implementation of obtaining the user's sequence representation is as follows: based on the Transformer encoder-decoder model, the interest representation sequence is used as the key and value, combined with the item representation to generate a decoding query, and finally the user's sequence representation is obtained through the output of the last query token.

7. The method for recommending multimodal alignment services based on prompt learning according to claim 6, characterized in that: The specific implementation process of step six is as follows: Sequence-item contrastive learning: Sequence-item contrastive learning is used as a pre-training task. The sequence representation u of user i obtained in step 6 is i Instead of actually buying items The similarity of , which measures the matching degree between user interests and target items, τ is a hyperparameter; Sequence-sequence contrastive learning: Using semantic invariance as a self-supervisory signal to enhance the robustness of sequence models; given u i and u′ i As the representation of the two enhanced sequences of the i-th user: i ,u′ i >Represents the similarity of two enhanced sequence representations of the same user, capturing the semantic invariance of user behavior; the training objectives are summarized as follows:​ Contrastive loss for cross-modal cue word alignment.

Images