Multi-modal commodity recommendation method based on large language model semantic enhancement

Abstract

The invention discloses a multi-modal commodity recommendation method based on large language model semantic enhancement, and the method employs LightGCN to carry out message propagation, obtains user ID embedding and commodity ID embedding, employs a cross attention mechanism to learn the relevance between modals, solves a problem that the semantic space between the modals is not matched, achieves the alignment of modal features, and improves the recommendation efficiency. According to the method, the multi-modal information is matched with a semantic space of a large language model by using an adapter to enhance information association between modals, and the understanding of the multi-modal information is enhanced by using the large language model, so that the problem of unbalance of the modal information is solved, and user modal embedding and commodity modal embedding are obtained; and finally, fusing the user ID embedding and the user modal embedding to obtain a final user embedding, fusing the commodity ID embedding and the commodity modal embedding to obtain a final commodity embedding, performing inner product operation on the final user embedding and the final commodity embedding to obtain a preference score, and sorting the commodities according to the preference score to obtain a recommendation list.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal commodity recommendation method based on semantic enhancement of a large language model. Background Art

[0002] With the rapid development of mobile Internet, a large number of products and information are displayed on the Internet, and it is difficult for users to decide which products and information are useful and pick out their favorite ones. The emergence of recommendation systems solves the problem of information overload. It predicts the user's ratings or preferences for all products, and recommends the most likely and relevant products that users may choose based on historical interaction data and the public. The significance of recommendation systems can be divided into two aspects. From the user's perspective: the recommendation system solves the problem of how users can efficiently obtain information of interest in the case of "information overload". From the company's perspective: the recommendation system solves the problem of how products can attract users, retain users, and increase user stickiness to the greatest extent, thereby achieving the company's revenue goals.

[0003] In the existing technology, the accuracy of product recommendation is not very high, and a lot of interaction between users and products is required to make more accurate recommendations. Users and products with little or no interaction will affect the accuracy of recommendations. In order to alleviate the data sparsity problem and the cold start problem, multimodal information is introduced into the recommendation system. Multimodal recommendation uses auxiliary multimodal information to supplement historical user and product interactions to improve recommendation performance. Multimodal models can represent and discover hidden relationships between different modalities, and may recover supplementary information that cannot be captured by single-modal methods and implicit interactions. However, although the existing multimodal product recommendation system has made significant progress in personalized recommendations and user experience, there are still some problems and challenges: 1. Insufficient association learning between modalities: Association and interaction learning between different modalities are crucial to understanding user preferences, but due to the huge differences in the feature spaces of different modalities, how to effectively integrate and learn information between different modalities becomes the key. 2. Imbalance of modal information: In some cases, some modalities may have much more information than other modalities, such as image information will be much more than text information. This leads to an imbalance between modalities, affecting the fairness and diversity of recommendation results. Summary of the invention

[0004] To this end, the purpose of the present invention is to provide a multimodal product recommendation method based on semantic enhancement of a large language model to solve the problems of insufficient association learning between modalities and imbalance of modal information in the prior art.

[0005] A multimodal product recommendation method based on semantic enhancement of a large language model comprises the following steps: Step S1, read the user and product shopping database, build the interaction graph of users and products, use multi-layer LightGCN as the propagation function on the interaction graph to capture high-order connections, and use the average function to merge the embedding of each layer of LightGCN to obtain the user ID embedding and product ID embedding; Step S2, read the commodity shopping database, obtain the text and picture information of the commodity, input the text and picture information of the commodity into the pre-trained model to generate the initial text modal features and initial picture modal features of the commodity, and use the projection function to project the initial text modal features and initial picture modal features of the commodity into a unified dimension. In addition, the initial text modal features of the user are obtained by aggregating the initial text modal features of the commodity, and the initial picture modal features of the user are obtained by aggregating the initial picture modal features of the commodity, and then the initial text modal features and initial picture modal features of the commodity are aligned by a cross-attention mechanism to obtain aligned commodity modal features, and the initial text modal features and initial picture modal features of the user are aligned by a cross-attention mechanism to obtain aligned user modal features, and the aligned commodity modal features and aligned user modal features are input into the large language model to perform feature enhancement of multimodal semantic information to obtain user modal embedding and commodity modal embedding; In step S3, the user ID embedding and the user modality embedding are fused to obtain the final user embedding, and the product ID embedding and the product modality embedding are fused to obtain the final product embedding. An inner product operation is performed on the final user embedding and the final product embedding to obtain a preference score. The products are sorted according to the preference score to obtain a personalized recommendation list, thereby realizing multimodal product recommendation.

[0006] Beneficial effects of the present invention: The present invention uses LightGCN for message propagation to obtain user ID embedding and product ID embedding, and then adopts the cross-attention mechanism to learn the correlation between modalities to solve the problem of semantic space mismatch between modalities, realize modal feature alignment to enhance the information association between modalities, and use an adapter to adapt multimodal information to the semantic space of a large language model, and adopt a large language model to enhance the understanding of multimodal information to solve the problem of modal information imbalance, obtain user modal embedding and product modal embedding, and finally fuse the user ID embedding and the user modal embedding to obtain the final user embedding, fuse the product ID embedding and the product modal embedding to obtain the final product embedding, perform inner product operation on the final user embedding and the final product embedding to obtain preference scores, sort the products according to the preference scores, obtain a personalized recommendation list, and realize multimodal product recommendation. By deeply analyzing and mining the user's semantic information and points of interest, the user's needs and preferences can be understood more accurately, thereby providing recommendation services that better meet user expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the process flow of the multimodal product recommendation method based on semantic enhancement of a large language model provided by the present invention. DETAILED DESCRIPTION

[0008] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0009] See also Figure 1 The present invention provides a multimodal product recommendation method based on semantic enhancement of a large language model, the method comprising steps S1 to S3: Step S1, read the user and product shopping database, build the interaction graph of users and products, use multi-layer LightGCN as the propagation function on the interaction graph to capture high-order connections, and use the average function to merge the embedding of each layer of LightGCN to obtain user ID embedding and product ID embedding.

[0010] Wherein, step S1 specifically includes: Step S11, read the user and product shopping database, build the user and product interaction graph, each user and product has an initial ID embedding, and the user and product interaction is represented as a matrix , where if the user u With goods i If there is interaction, Elements in r u,i =1, otherwise r u,i = 0, u ∈U, i ∈I, U is the user set, I is the product set; In step S12, multi-layer LightGCN is used to propagate messages on the interaction graph to capture high-order connections. The expression is as follows: ; in, It is l The user ID embedding and product ID embedding output by the LightGCN layer, Aggregate is the message propagation function, It is l-1 layer of LightGCN outputs user ID embedding and product ID embedding, A is the adjacency matrix constructed by the interaction matrix, D is the diagonal matrix of A, where the initial embedding matrix , It is the initial ID embedding of the user and the initial ID embedding of the item on the interaction graph; Step S13, use the average function to merge the embeddings in all LightGCN layers to obtain the user ID embedding and product ID embedding, expressed as follows: ; in, is the user ID embedding and product ID embedding, Mean is the average function, It is the user ID embedding and product ID embedding output by the first layer of LightGCN. It is the user ID embedding and product ID embedding output by the second layer LightGCN. The user ID is embedded in And product ID embed .

[0011] Step S2, read the product shopping database to obtain the text and image information of the product, input the text and image information of the product into the pre-trained model to generate the initial text modal features and initial image modal features of the product, and use the projection function to project the initial text modal features and initial image modal features of the product into a unified dimension. In addition, the initial text modal features of the user are obtained by aggregating the initial text modal features of the product, and the initial image modal features of the user are obtained by aggregating the initial image modal features of the product. Then, the initial text modal features and initial image modal features of the product are aligned using a cross-attention mechanism to obtain aligned product modal features, and the initial text modal features and initial image modal features of the user are aligned using a cross-attention mechanism to obtain aligned user modal features. The aligned product modal features and aligned user modal features are input into the large language model to perform feature enhancement of multimodal semantic information to obtain user modal embedding and product modal embedding.

[0012] Wherein, step S2 specifically includes: Step S21, read the product shopping database, obtain the text and image information of the product, input the text and image information of the product into the pre-trained model to generate the initial text modal features and initial image modal features of the product. The initial modal text and image features of the product are generated by the BERT and ResNet pre-trained models. The two modalities have different dimensions. Considering the semantic differences between the modalities, the projection function is used to project the initial text modal features and initial image modal features of the product into a unified dimension. The expression is as follows: ; ; in, It is a commodity i The text features after unified dimension, commodity i The initial text modality features of Encoder are projection functions parameterized by the transformation matrix. is the text transformation matrix, It is a commodity i The image features after unified dimension, It is a commodity i The initial image modality features of is the image transformation matrix; Step S22: Due to privacy issues, it is difficult to obtain the user's image and text information. Existing methods often reuse the user ID embedding as the input of the modality-specific graph, resulting in the coupling of the collaboration signal and the modality signal. In the present invention, the user's initial text modality feature is obtained by aggregating the initial text modality feature of the product, and the user's initial image modality feature is obtained by aggregating the initial image modality feature of the product. The expression is as follows: ; ; in, Is a user u The initial text features, Is a user u The initial image features, Indicates user u The set of neighbors on the user and item interaction graph G; this operation ensures that the ID embedding and modality features are updated separately; Step S23, in order to capture the correlation between the image and the text and solve the problem of cross-modal information mismatch between the image and the text. After obtaining the initial text and image modal features of the product, the initial text modal features and the initial image modal features of the product are aligned using a cross-attention mechanism to obtain aligned product modal features, and the initial text modal features and the initial image modal features of the user are aligned using a cross-attention mechanism to obtain aligned user modal features, so as to improve the effect of information fusion, and the expression is as follows: ; ; in, , , They are the query, key, and value matrices of the cross-attention of the product. , , are the query, key, and value matrices of user cross-attention, respectively. It is the product modal embedding after the text and image are aligned. It is the user modal embedding after the text and image are aligned; S24, in order to enhance the natural understanding of multimodal features by large language models, an adapter is used to align the multimodal feature space with the input space of the large language model. The expression is as follows: ; ; in, is the item embedding aligned with the large language model, is a user embedding aligned with a large language model. The adapter is implemented by a multi-layer perceptron MLP. , It is the parameter that needs to be adjusted during training; S25. In practical applications, users’ multimodal data is often unbalanced. For example, some users prefer text data, while others prefer image data. This adds difficulty to the semantic understanding of multimodal data. The large language model processes multimodal information in a comprehensive and comprehensive manner. It can obtain rich information from multiple modalities and improve model performance and robustness. In addition, traditional unimodal processing methods may lead to the loss of some information, while large language models can capture more relevant information in multimodal data and reduce the possibility of information loss. Therefore, the present invention adopts a text-based open source large language model LLM (such as Vicuna2 and Llama-3) to enhance the semantics of multimodal information. LLM takes the representation of different modalities as input, and performs semantic understanding and reasoning on the input to obtain user modal embedding and product modal embedding. The expressions are as follows: ; ; in, It is the modal embedding of the product after LLM semantic enhancement. It is the user's modal embedding after LLM semantic enhancement. The modal embedding of all users is represented as the user modal embedding matrix , the modal embedding of all products is represented as the product modal embedding matrix .

[0013] In step S3, the user ID embedding and the user modality embedding are fused to obtain the final user embedding, and the product ID embedding and the product modality embedding are fused to obtain the final product embedding. An inner product operation is performed on the final user embedding and the final product embedding to obtain a preference score. The products are sorted according to the preference score to obtain a personalized recommendation list, thereby realizing multimodal product recommendation.

[0014] Wherein, step S3 specifically includes: Step S31, the user ID embedding and the user modality embedding are fused to obtain the final user embedding, and the product ID embedding and the product modality embedding are fused to obtain the final product embedding, which is expressed as follows: ; ; Among them, Norm is a normalization function used to alleviate the scale difference between embeddings. is an adjustable factor that controls the modal embedding integral, is the end user embed, It is the ultimate commodity embedding; Step S32, use the inner product to calculate the user u About Products i Preference score , using Bayesian personalized ranking BPR as the loss function to optimize the model parameters, the expression is as follows: ; in, is the total loss, R is a set of triplets used for training, represents the positive sample items that the user has interacted with, Indicates negative sample items that the user has not interacted with. is the sigmoid function, Represents the user predicted by the model u Check sample products Ratings, Represents the user predicted by the model u Check sample products Ratings, is the regularization coefficient, are all the learnable parameters that make up the model The L2 norm of .

[0015] According to the multimodal product recommendation method based on semantic enhancement of a large language model provided by the present invention, LightGCN is used for message propagation to obtain user ID embedding and product ID embedding, and then a cross-attention mechanism is used to learn the correlation between modalities to solve the problem of semantic space mismatch between modalities and realize modal feature alignment to enhance the information association between modalities. An adapter is used to adapt multimodal information to the semantic space of a large language model, and a large language model is used to enhance the understanding of multimodal information to solve the problem of modal information imbalance, to obtain user modal embedding and product modal embedding, and finally, the user ID embedding and the user modal embedding are fused to obtain the final user embedding, and the product ID embedding and the product modal embedding are fused to obtain the final product embedding. An inner product operation is performed on the final user embedding and the final product embedding to obtain a preference score, and the products are sorted according to the preference score to obtain a personalized recommendation list to realize multimodal product recommendation. By performing in-depth analysis and mining of the user's semantic information and points of interest, the user's needs and preferences can be understood more accurately, thereby providing a recommendation service that better meets the user's expectations.

[0016] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0017] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A multimodal product recommendation method based on semantic enhancement of a large language model, characterized in that: The following steps are involved: Step S1, read the user and product shopping database, build the interaction graph of users and products, use multi-layer LightGCN as the propagation function on the interaction graph to capture high-order connections, and use the average function to merge the embedding of each layer of LightGCN to obtain the user ID embedding and product ID embedding; Step S2, read the commodity shopping database, obtain the text and picture information of the commodity, input the text and picture information of the commodity into the pre-trained model to generate the initial text modal features and initial picture modal features of the commodity, and use the projection function to project the initial text modal features and initial picture modal features of the commodity into a unified dimension. In addition, the initial text modal features of the user are obtained by aggregating the initial text modal features of the commodity, and the initial picture modal features of the user are obtained by aggregating the initial picture modal features of the commodity, and then the initial text modal features and initial picture modal features of the commodity are aligned by a cross-attention mechanism to obtain aligned commodity modal features, and the initial text modal features and initial picture modal features of the user are aligned by a cross-attention mechanism to obtain aligned user modal features, and the aligned commodity modal features and aligned user modal features are input into the large language model to perform feature enhancement of multimodal semantic information to obtain user modal embedding and commodity modal embedding; In step S3, the user ID embedding and the user modality embedding are fused to obtain the final user embedding, and the product ID embedding and the product modality embedding are fused to obtain the final product embedding. An inner product operation is performed on the final user embedding and the final product embedding to obtain a preference score. The products are sorted according to the preference score to obtain a personalized recommendation list, thereby realizing multimodal product recommendation.

2. The multimodal product recommendation method based on semantic enhancement of a large language model according to claim 1, characterized in that: Step S1 specifically includes: Step S11, read the user and product shopping database, build the user and product interaction graph, and the user and product interaction is represented as a matrix , where if the user u With goods i If there is interaction, Elements in r u,i =1, otherwise r u,i = 0, u ∈U, i ∈I, U is the user set, I is the product set; In step S12, multi-layer LightGCN is used to propagate messages on the interaction graph to capture high-order connections. The expression is as follows: ； in, It is l The user ID embedding and product ID embedding output by the LightGCN layer, Aggregate is the message propagation function, It is l -1 layer of LightGCN outputs user ID embedding and product ID embedding, A is the adjacency matrix constructed by the interaction matrix, D is the diagonal matrix of A, where the initial embedding matrix , It is the initial ID embedding of the user and the initial ID embedding of the item on the interaction graph; Step S13, use the average function to merge the embeddings in all LightGCN layers to obtain the user ID embedding and product ID embedding, expressed as follows: ； in, is the user ID embedding and product ID embedding, Mean is the average function, It is the user ID embedding and product ID embedding output by the first layer of LightGCN. It is the user ID embedding and product ID embedding output by the second layer LightGCN. The user ID is embedded in And product ID embed .

3. The multimodal product recommendation method based on semantic enhancement of a large language model according to claim 2 is characterized in that: Step S2 specifically includes: Step S21, read the product shopping database, obtain the text and picture information of the product, input the text and picture information of the product into the pre-trained model to generate the initial text modal features and initial picture modal features of the product, and use the projection function to project the initial text modal features and initial picture modal features of the product into a unified dimension, the expression is as follows: ； ； in, It is a commodity i The text features after unified dimension, commodity i The initial text modality features of Encoder are projection functions parameterized by the transformation matrix. is the text transformation matrix, It is a commodity i The image features after unified dimension, It is a commodity i The initial image modality features of is the image transformation matrix; Step S22, the initial text modal features of the user are obtained by aggregating the initial text modal features of the products, and the initial image modal features of the user are obtained by aggregating the initial image modal features of the products. The expression is as follows: ； ； in, Is a user u The initial text features, Is a user u The initial image features, Indicates user u The set of neighbors on the user and product interaction graph G; Step S23, using a cross attention mechanism to align the initial text modal features and the initial image modal features of the product to obtain aligned product modal features, and using a cross attention mechanism to align the initial text modal features and the initial image modal features of the user to obtain aligned user modal features, the expression is as follows: ； ； in, , , They are the query, key, and value matrices of the cross-attention of the product. , , are the query, key, and value matrices of user cross-attention, respectively. It is the product modal embedding after the text and image are aligned. It is the user modal embedding after the text and image are aligned; S24, uses an adapter to align the multimodal feature space with the input space of the large language model, expressed as follows: ； ； in, is the item embedding aligned with the large language model, is a user embedding aligned with a large language model. The adapter is implemented by a multi-layer perceptron MLP. , It is the parameter that needs to be adjusted during training; S25 uses the text-based open source large language model LLM to enhance the semantics of multimodal information. LLM takes the representation of different modalities as input, and performs semantic understanding and reasoning on the input to obtain user modal embedding and product modal embedding. The expressions are as follows: ； ； in, It is the modal embedding of the product after LLM semantic enhancement. It is the user's modal embedding after LLM semantic enhancement. The modal embedding of all users is represented as the user modal embedding matrix , the modal embedding of all products is represented as the product modal embedding matrix .

4. The multimodal product recommendation method based on semantic enhancement of a large language model according to claim 3 is characterized in that: Step S3 specifically includes: Step S31, the user ID embedding and the user modality embedding are fused to obtain the final user embedding, and the product ID embedding and the product modality embedding are fused to obtain the final product embedding, which is expressed as follows: ； ； Among them, Norm is a normalization function used to alleviate the scale difference between embeddings. is an adjustable factor that controls the modal embedding integral, is the end user embed, It is the ultimate commodity embedding; Step S32, use the inner product to calculate the user u About Products i Preference score , using Bayesian personalized ranking BPR as the loss function to optimize the model parameters, the expression is as follows: ； in, is the total loss, R is a set of triplets used for training, represents the positive sample items that the user has interacted with, Indicates negative sample items that the user has not interacted with. is the sigmoid function, Represents the user predicted by the model u Check sample products Rating, Represents the user predicted by the model u Check sample products Rating, is the regularization coefficient, are all the learnable parameters that make up the model The L2 norm of .

Images