Multimodal recommendation method and system based on multimodal prompt and difficult modal enhancement

Abstract

The application belongs to the field of multi-modal recommendation, and provides a multi-modal recommendation method and system based on multi-modal prompts and difficult modal enhancement, and the technical scheme is as follows: a multi-modal prompt component is designed to model different specific modal user interests by using a user representation and different modal prompt representations, which can effectively and efficiently learn user interests, thereby improving the accuracy of the recommendation system; a difficult modal enhancement component is designed to enhance user interest learning in difficult modal, and ensure that user interests under all modes are well learned, thereby improving the accuracy of the recommendation system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal recommendation, and particularly relates to a user interest learning method and system based on multi-modal prompt and difficult modal enhancement. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] A recommendation system can recommend goods according to user preferences, which has attracted the research interest of many researchers. In recent years, considering the multi-modal data of goods, such as product pictures and text, which can reflect user interests and product features from different angles, many researchers have begun to study multi-modal recommendation systems.

[0004] Existing multi-modal recommendation systems first extract modal features from multi-modal data of goods, and then use the results of modal fusion to assist user-goods interaction data to predict user interest scores for goods.

[0005] The inventors have found that the existing multi-modal recommendation system has the following technical problems:

[0006] (1) Existing recommendation systems usually represent each user as multiple different representations to learn different specific modal user interests, which ignores the potential connection between different specific modal user interests. In fact, since different modal data of goods reflects product features from different angles, there may be consistency information between user interests for different modalities. Even if some research works design additional models to learn this consistency information, they significantly increase the memory occupation and computational cost of the model, and still do not achieve enough performance improvement.

[0007] (2) Existing methods usually use BPR mechanism to supervise the final user interest learning, i.e., making the final interest score of the user for the positive example goods (the result after fusing the specific modal user scores) greater than the final interest for the negative example goods. But this method cannot optimize the learning of each specific modal user interest. Therefore, there may be a user interest in a certain modality that is not well learned (called difficult modality), which leads to the predicted specific modal user score being unreliable, thereby affecting the final prediction of the user's final interest in the goods. SUMMARY

[0008] To solve at least one of the technical problems in the background art, the present application provides a multi-modal recommendation method and system based on multi-modal prompt and difficult modal enhancement, which designs a multi-modal prompt component and a difficult modal enhancement component to better learn user interests and improve the accuracy of the recommendation system.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A first aspect of the present invention provides a multimodal recommendation method based on multimodal prompts and hard mode enhancement, comprising the following steps:

[0011] Acquire user data and product multimodal data;

[0012] A multimodal recommendation model is trained based on user data and multimodal product data; wherein the training process of the multimodal recommendation model includes:

[0013] Learning multimodal representations of users based on user data involves modeling user interests in different modalities using a single user representation and different modal cue representations, thus obtaining a multimodal representation of the user.

[0014] Based on multimodal data of goods, multimodal representations of positive and negative examples of goods are extracted respectively. Difficult modes are identified based on the multimodal representations of positive and negative examples of goods. Multimodal representations of difficult negative examples of goods are constructed based on the difficult modes.

[0015] By combining user multimodal representations, positive example product multimodal representations, negative example product multimodal representations, and difficult negative example product multimodal representations, we can predict users' interest scores for positive example products, negative example products, and difficult negative example products.

[0016] The training objective is to make the user's interest score for positive example items greater than the user's interest score for negative example items and difficult negative example items;

[0017] Based on the trained multimodal recommendation model, the system outputs a score of the user's interest in the predicted item.

[0018] A second aspect of the present invention provides a multimodal recommendation system based on multimodal prompts and hard modality enhancement, comprising:

[0019] The data acquisition module is used to acquire user data and multimodal product data;

[0020] A multimodal recommendation model training module is used to train a multimodal recommendation model based on user data and product multimodal data; wherein, the training process of the multimodal recommendation model includes:

[0021] Learning multimodal representations of users based on user data involves modeling user interests in different modalities using a single user representation and different modal cue representations, thus obtaining a multimodal representation of the user.

[0022] Based on multimodal data of goods, multimodal representations of positive and negative examples of goods are extracted respectively. Difficult modes are identified based on the multimodal representations of positive and negative examples of goods. Multimodal representations of difficult negative examples of goods are constructed based on the difficult modes.

[0023] By combining user multimodal representations, positive example product multimodal representations, negative example product multimodal representations, and difficult negative example product multimodal representations, the system predicts user interest scores for positive example products, negative example products, and difficult negative example products. The training objective is to ensure that user interest scores for positive example products are greater than user interest scores for negative example products and difficult negative example products.

[0024] The recommendation module is used to output a user's interest score for the item to be predicted based on the trained multimodal recommendation model.

[0025] A third aspect of the present invention provides a computer-readable storage medium.

[0026] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the multimodal recommendation method based on multimodal cues and hard modality enhancement as described in the first aspect.

[0027] A fourth aspect of the present invention provides a computer device.

[0028] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the multimodal recommendation method based on multimodal prompting and hard modality enhancement as described in the first aspect.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] (1) The multimodal cue component designed in this invention uses a user representation and different modal cue representations to model different specific modal user interests, which can save model space while ensuring the accuracy of the recommendation system.

[0031] (2) The difficult modality enhancement component designed in this invention can enhance the learning of user interests in difficult modalities, ensuring that user interests in all modalities are well learned, thereby improving the accuracy of the recommendation system.

[0032] (3) The two components designed in this invention can be easily applied to existing multimodal recommendation methods, helping them improve recommendation accuracy while reducing memory consumption.

[0033] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0034] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0035] Figure 1 This is a flowchart of the multimodal prompting interest learning component provided in an embodiment of the present invention;

[0036] Figure 2 This is a flowchart of the difficult mode enhancement component provided in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] As mentioned in the background section of this invention, in order to address how to efficiently model user interests across different modalities and how to identify modalities where user interests are difficult to learn, this invention designs a multimodal cueing component that models user interests across different specific modalities using a single user representation and different modal cueing representations. This enables the effective and efficient learning of user interests, thereby improving the accuracy of the recommendation system.

[0041] Example 1

[0042] This embodiment provides a multimodal recommendation method based on multimodal prompts and hard modality enhancement, which includes two components: a multimodal prompting learning component and a hard modality enhancement component.

[0043] The multimodal prompting learning component takes a user and a product as input and outputs a user's interest score for the product. The flowchart is as follows: Figure 1 As shown.

[0044] The specific steps of this component are described in detail below with reference to an example. First, the following notation is explained. Assume there is a user set U, a product set I, and a product modality set M. Each user u∈U has a historical interaction product set I. u Each product i in the product set I has its corresponding modal characteristics. in Let m be the feature of the m-th modality of product i.

[0045] The specific steps for using the multimodal prompt component are as follows:

[0046] (1) Multimodal representation learning of goods

[0047] This invention extracts multimodal representations of goods from multimodal data of goods and maps different modal features of goods to a unified latent space.

[0048] In this embodiment, a multilayer perceptron model is used to integrate each modal feature. Mapped to a d-dimensional representation The calculation formula is as follows:

[0049]

[0050] in, Let m be the modal representation of product i. and These are the parameters of the multilayer perceptron model. tanh() is the nonlinear activation function. The multimodal representation of the commodity is e. i ∈R d×|M| It is composed of all modal features, as follows:

[0051]

[0052] (2) User multimodal representation learning

[0053] This embodiment utilizes a basic user representation and different modal cues to construct a multimodal user representation and learn the interests of users with different specific modalities.

[0054] First, assign a d-dimensional representation f to each user u. u This serves as a basic user representation.

[0055] Subsequently, a small number of modality-specific cue representations are assigned to each modality m. Where Q represents the number of cues in each modality, and P represents the modal cues. m Enhanced basic user representation f u It can represent the user's interest in the m-th modality.

[0056] In this way, the shared base user representation will model the consistency of user interests across different modalities, while the modal cue representation will model their differences.

[0057] It is worth noting that the number of modalities is much smaller than the number of users; therefore, the number of parameters introduced by assigning a cue representation to each modality is much smaller than the number of parameters introduced by assigning multiple representations to each user. Ultimately, the user multimodal representation e u ∈R d×|M| It is composed of a base user representation enhanced by all modal cues, and the calculation formula is as follows.

[0058]

[0059] in, The sum() function represents the user's interest representation in the m-modal and performs element-level summation on the input.

[0060] (3) Modeling of user-product interaction relationship

[0061] Based on multimodal representations of users and products, the user-product interaction relationship modeling method F predicts the user interest score for a specific modality, and combines all the user interest scores for a specific modality to obtain the final user interest score for the product.

[0062] Therefore, the formula for calculating a user's interest score in a product is as follows:

[0063]

[0064] Among them, y u,i This represents the final interest score of user u for product i. This represents the user interest score for product i in a specific modality under modality m, representing user u's interest in the product i.

[0065] (4) Recommended target loss

[0066] This embodiment uses BPR loss as the recommended target.

[0067] First, construct the training set ω = {(u,i,k) | u∈U,i∈I}. u ,k∈I\I u}, where u is a user in the user set U, i∈I u Let k ∈ I be the positive example items (historical interaction items) for user u. u Let each item be a negative example for user u (randomly sampled from items the user has not interacted with). The training triple (u, i, k) represents user u's preference for item i over item k. Subsequently, the recommendation target loss is defined as follows:

[0068]

[0069] Among them, y u,i and y u,k These are the interest scores of user u for product i and product k, respectively.

[0070] By optimizing the loss function, users will have a higher interest score for products they like than for products they dislike.

[0071] 2. Difficult Mode Enhancement Component

[0072] It is worth noting that the aforementioned recommendation objective loss can only supervise the learning of end-user interests and cannot optimize the learning of user interests for each specific modality. Therefore, there may be user interests in a certain modality that are not learned well (referred to as a difficult modality), resulting in unreliable predicted user scores for that specific modality, thus affecting the prediction of the final user's interest in the product.

[0073] To mitigate this problem, the present invention designs a difficult modality enhancement component. First, the difficult modality is identified, and second, user interest learning in the difficult modality is enhanced.

[0074] The flowchart of the difficult mode enhancement component is as follows: Figure 2 As shown, the specific steps are as follows:

[0075] (1) Confirmation of difficult modes

[0076] Intuitively, given a training triple (u,i,k), if the difference in interest scores between positive example item i and negative example item k is small in the m-th modality, it means that the model has difficulty in judging whether the user likes or dislikes items based on the m-th modality. That is, the user's specific modal interests in the m-th modality have not been learned well.

[0077] Therefore, for each training triple (u,i,k), the difference in interest scores between user u for positive item i and negative item k is calculated across all modalities. The difficult modality is defined as the modality with the smallest score difference, and the calculation formula is as follows.

[0078]

[0079] in, This is the index of the difficult mode in the triple (u,i,k). and These are the interest scores of user u for positive example item i and negative example item k in the m-th modality, respectively.

[0080] (2) Enhanced learning of interest among users with difficult modalities

[0081] To enhance user interest learning in difficult modalities, this embodiment constructs a difficult negative example item k for each triple (u,i,k). *The difficulty of the negative example product differs from that of the positive example product only in the difficulty mode.

[0082] Difficult negative example product k * The multimodal representation of is defined as follows.

[0083]

[0084] Subsequently, based on the user's multimodal representation e u Multimodal representation of difficult negative examples of goods e k* The user u obtains the negative example of product k with difficulty. * Interest score y u,k* The calculation formula is defined as follows:

[0085] y u,k* =F(e u ,e k* ).

[0086] The purpose of hard modal regularization is to increase the interest score of user u for positive example item i to that for hard negative example item k. * The interest score is calculated using the following formula.

[0087]

[0088] Among them, y u,i and y u,k* For user u, product i and product k are respectively. * Interest score.

[0089] Since the constructed difficult negative example goods differ from the positive example goods only in their modal representation within the difficult mode, it is guaranteed that user u's interest score for the positive example goods i is greater than that for the difficult negative example goods k. * Interest scores enable users to focus their interest learning on difficult modalities, thereby enhancing their interest learning in difficult modalities.

[0090] Finally, the overall loss function of the proposed technology is defined as follows:

[0091]

[0092] Among them, L bpr and L weak These represent the recommendation loss and the hard mode regularization, respectively. α is a balance parameter that adjusts the importance of both, Θ is the set of parameters for the technique, β is the weight for parameter regularization, and the last term is used to prevent overfitting.

[0093] Example 2

[0094] This embodiment provides a multimodal recommendation system based on multimodal prompts and hard modality enhancement, including:

[0095] The data acquisition module is used to acquire user data and multimodal product data;

[0096] A multimodal recommendation model training module is used to train a multimodal recommendation model based on user data and product multimodal data; wherein, the training process of the multimodal recommendation model includes:

[0097] Learning multimodal representations of users based on user data involves modeling user interests in different modalities using a single user representation and different modal cue representations, thus obtaining a multimodal representation of the user.

[0098] Based on multimodal data of goods, multimodal representations of positive and negative examples of goods are extracted respectively. Difficult modes are identified based on the multimodal representations of positive and negative examples of goods. Multimodal representations of difficult negative examples of goods are constructed based on the difficult modes.

[0099] By combining user multimodal representations, positive example product multimodal representations, negative example product multimodal representations, and difficult negative example product multimodal representations, the system predicts user interest scores for positive example products, negative example products, and difficult negative example products. The training objective is to ensure that user interest scores for positive example products are greater than user interest scores for negative example products and difficult negative example products.

[0100] The recommendation module is used to output a user's interest score for the item to be predicted based on the trained multimodal recommendation model.

[0101] Example 3

[0102] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the multimodal recommendation method based on multimodal prompts and hard modality enhancement as described above.

[0103] Example 4

[0104] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal recommendation method based on multimodal prompts and hard modality enhancement as described above.

[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal recommendation method based on multimodal prompts and hard modality enhancement, characterized in that, Includes the following steps: Acquire user data and product multimodal data; A multimodal recommendation model is trained based on user data and multimodal product data; wherein the training process of the multimodal recommendation model includes: User multimodal representation learning is performed based on user data. This involves modeling user interests across different modalities using a single user representation and various modal cue representations, resulting in a multimodal user representation. Specifically, this modeling includes: For each user Assign one Wei said As a basic user representation; For each mode Assigning specific modal hints ,in The number of cues represented in each modality; Modal cues will be represented Enhanced basic user representation Indicates the user's opinion on the first Interest in each modality; The shared base user representation will model the consistency of user interests across different modalities, while the modal cue representation will model their differences; Based on multimodal data of goods, multimodal representations of positive and negative examples of goods are extracted respectively. Difficult modes are identified based on these multimodal representations. Multimodal representations of difficult negative examples of goods are then constructed based on these difficult modes. The identification of difficult modes based on the multimodal representations of positive and negative examples of goods includes: Calculate the difference in interest scores between users for positive and negative products across all modalities, and designate the modality with the smallest score difference as the difficult modality. By combining user multimodal representations, positive example product multimodal representations, negative example product multimodal representations, and difficult negative example product multimodal representations, the system predicts user interest scores for positive example products, negative example products, and difficult negative example products. The training objective is to ensure that user interest scores for positive example products are greater than user interest scores for negative example products and difficult negative example products. Based on the trained multimodal recommendation model, the system outputs a score of the user's interest in the predicted item.

2. The multimodal recommendation method based on multimodal prompts and hard modality enhancement as described in claim 1, characterized in that, Assigning a cue to each modality introduces far fewer parameters than assigning multiple representations to each user.

3. The multimodal recommendation method based on multimodal prompts and hard mode enhancement as described in claim 1, characterized in that, After extracting the multimodal representations of positive and negative examples, the resulting multimodal representations are mapped to a unified latent space.

4. The multimodal recommendation method based on multimodal prompts and hard modality enhancement as described in claim 1, characterized in that, When predicting users' interest scores for positive, negative, and difficult negative products, a user-product interaction relationship modeling method is used.

5. The multimodal recommendation method based on multimodal prompts and hard mode enhancement as described in claim 1, characterized in that, The loss function during the training of the multimodal recommendation model is: in, and These are recommendation loss and hard mode regularization, respectively. To adjust the balance parameters of the two ratios, A set of parameters for the technology. The weights are used for parameter regularization.

6. A multimodal recommendation system based on multimodal prompts and hard modality enhancement, characterized in that, include: The data acquisition module is used to acquire user data and multimodal product data; A multimodal recommendation model training module is used to train a multimodal recommendation model based on user data and product multimodal data; wherein, the training process of the multimodal recommendation model includes: User multimodal representation learning is performed based on user data. This involves modeling user interests across different modalities using a single user representation and various modal cue representations, resulting in a multimodal user representation. Specifically, this modeling includes: For each user Assign one Wei said As a basic user representation; For each mode Assigning specific modal hints ,in The number of cues represented in each modality; Modal cues will be represented Enhanced basic user representation Indicates the user's opinion on the first Interest in each modality; The shared base user representation will model the consistency of user interests across different modalities, while the modal cue representation will model their differences; Based on multimodal data of goods, multimodal representations of positive and negative examples of goods are extracted respectively. Difficult modes are identified based on these multimodal representations. Multimodal representations of difficult negative examples of goods are then constructed based on these difficult modes. The identification of difficult modes based on the multimodal representations of positive and negative examples of goods includes: Calculate the difference in interest scores between users for positive and negative products across all modalities, and designate the modality with the smallest score difference as the difficult modality. By combining user multimodal representations, positive example product multimodal representations, negative example product multimodal representations, and difficult negative example product multimodal representations, the system predicts user interest scores for positive example products, negative example products, and difficult negative example products. The training objective is to ensure that user interest scores for positive example products are greater than user interest scores for negative example products and difficult negative example products. The recommendation module is used to output a user's interest score for the item to be predicted based on the trained multimodal recommendation model.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal recommendation method based on multimodal prompts and hard modality enhancement as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal recommendation method based on multimodal prompts and hard modality enhancement as described in any one of claims 1-5.

Images