Multi-modal commodity recommendation system based on contrastive learning and model-based reinforcement learning

Abstract

The application discloses a multi-modal commodity recommendation system based on contrast learning and model-based reinforcement learning. First, multi-modal data of commodity images, texts and labels in a recommendation system and user behavior data collected are obtained, and commodity features are constructed through an unsupervised learning method mainly based on contrast learning; then, user features are constructed as states by using the constructed commodity features and the user behavior data, a recommended commodity is defined as an action, and user feedback is defined as an environmental reward, and an offline reinforcement learning model is trained; finally, the model is used as a pre-training model and is deployed to a real scene, and when new user behavior data is collected by the system, the model is updated in time. The application uses a framework of reinforcement learning, applies contrast learning multi-modal tasks, and comprehensively constructs a Markov process of user interaction with a recommendation system by using a large amount of commodity and user shopping information, so that the accuracy of the recommendation system is improved.

Description

Technical Field

[0001] This invention relates to a multimodal product recommendation system based on contrastive learning and model-based reinforcement learning, belonging to the technical field of product recommendation systems, and involving multimodal representation learning, contrastive learning, reinforcement learning and other fields. Background Technology

[0002] The application of reinforcement learning in recommender systems primarily addresses the challenges posed by the increasing abundance of information resources and the diversification of user preferences in traditional recommendation algorithms. Traditional methods have limitations, such as overly simplistic recommendation results and the inability to handle cold starts. Reinforcement learning, as a learnable policy selection method, offers advantages such as adaptability, dynamism, and strong learning capabilities. With the rapid development of information technologies such as mobile internet, social networks, and e-commerce platforms, the scale and complexity of data are constantly increasing. Deep reinforcement learning can handle more complex datasets, and model-based reinforcement learning can effectively utilize already collected data to learn more stable and robust policies.

[0003] Multimodal problems have received relatively little research in recommender systems. With the development of the internet, the modal attributes of products have increased, and deep learning is often used to learn representations of each modality. Existing techniques for obtaining multimodal representations typically use direct concatenation or modality fusion methods. Direct concatenation of multiple modal features fails to consider the correlation between modalities, while existing modality fusion methods, although solving this problem, struggle to consider the correlation between products with the same label. Summary of the Invention

[0004] Purpose of the Invention: Traditional recommendation systems primarily rely on user behavior data, but suffer from data sparsity and other issues, leading to low recommendation accuracy. To address these problems and shortcomings, this invention provides a multimodal product recommendation system based on contrastive learning and model-based reinforcement learning. This invention utilizes contrastive learning to fuse modal features, associating products with the same labels by constructing positive and negative samples, learning effective task representations, and fusing multimodal information. By introducing multimodal data and contrastive learning techniques, this invention improves the accuracy and granularity of product similarity calculation, thereby enhancing the accuracy of product representation, while also leveraging the inherent semantic information of products. When new users enter the system, traditional recommendation systems struggle to provide personalized recommendations due to a lack of historical behavior data. This invention introduces reinforcement learning techniques, modeling the recommendation task as a partially observable Markov decision process, and using reinforcement learning algorithms for recommendations. This treats new users as initial states without past behavior, then makes recommendations based on the user's interaction with the system, thus achieving personalized recommendations for new users and fully reusing information from different users from the perspective of interaction behavior. Traditional recommendation systems primarily rely on user behavior data for recommendations, failing to fully leverage the multimodal information of products. This invention introduces multimodal data, integrating information such as images and text into the recommendation process, thereby enhancing the personalization of recommendations and improving user experience.

[0005] Due to task limitations, this method assumes that user behavior trajectory data will not be too long, as excessive length would result in limited dimensions of latent variables, making it impossible to fully characterize past historical information. This assumption is reasonable, as we consider each period from when the user opens the system to when they close it as the same trajectory data.

[0006] Specifically, the key to this invention is to use contrastive learning techniques to learn multimodal representations of goods, use partially observable reinforcement learning to model the recommendation system problem, use model-based reinforcement learning methods to learn the system model from the observation sequence, and use the model to construct a recommendation strategy.

[0007] Technical solution: A multimodal product recommendation system based on contrastive learning and model-based reinforcement learning, including a multimodal feature fusion module based on contrastive learning, an offline pre-training module for the recommendation system based on model-based reinforcement learning, and an online update module for the system to be deployed in real-world scenarios using reinforcement learning.

[0008] In the multimodal feature fusion module based on contrastive learning: information about recommended products is collected, including images, text descriptions, and tags; data augmentation is performed on the images, and contrastive learning is used to learn image representations; text representations of product descriptions are learned using contrastive learning methods; and multimodal representations of products with the same tags are learned using contrastive learning methods by combining image and text representations.

[0009] In the offline pre-training module of the recommendation system based on model-based reinforcement learning: the observations, states, actions, transition functions, and reward functions of partially observable reinforcement learning are defined; the collected user behavior data and product data are used as offline reinforcement learning trajectory data, and the observations are used to generate embeddings through the encoder. The hidden states are constructed through the embeddings, and the recurrent state space model is trained; the trained model is used to learn the policy using the actor-critic method.

[0010] The system is deployed in a real-world reinforcement learning online update module: For users in real-world scenarios, their data is encoded into a recurrent state space model, and product recommendations are given to users based on the results of the actors; using continuously acquired user behavior data, the data is used to construct the trajectory of the reinforcement learning model, thereby updating the recurrent state space model, the actor neural network, and the commentator neural network.

[0011] The specific implementation process of the multimodal feature fusion module based on contrastive learning is as follows:

[0012] Step 100: Extract product images, text descriptions, and tags from the platform to form a dataset;

[0013] Step 101: Construct an image feature extraction neural network; for each image, perform data augmentation, and train the network using the original image, the augmented image, and other images, employing information noise contrast estimation loss until convergence.

[0014] Step 102: Construct a text feature extraction neural network; for the text description of the product, perform data augmentation, using the original text, the data-augmented text, and other text, and train it using information noise contrast estimation loss; for the text description of the product, train it using a masked language model.

[0015] Step 103: Construct a neural network that fuses image and text features; using the image and text features extracted from the products in steps 101 and 102, input them into the fusion feature network, use the fusion features of other products with the same label as positive samples, and the fusion features of products with different labels as negative samples, and use information noise contrast estimation loss to train until convergence.

[0016] The specific implementation process of the offline pre-training module of the recommendation system based on model reinforcement learning is as follows:

[0017] Step 200: Define state S_T as the recommended products and user behavior data from time 1 to the current time T. The recommended products are represented by the data obtained in step 103. The observations are the recommended products and user behavior responses at the current time. The actions are the products recommended by the system to the user. The transition function is the new recommended product and user behavior sequence S_{T+1} reached after the current recommended action and the user's behavior response in the previous state S_T. The reward function is the numerical response of the user to the currently recommended product. Construct the above offline trajectory sequence from the existing user behavior data.

[0018] Step 201: Construct the encoder and decoder; construct a cyclic state-space model for modeling environmental changes;

[0019] Step 202: Use the offline trajectory sequence obtained in step 200 to train the encoder and decoder in step 201, as well as the cyclic state space model, to minimize the reconstruction loss, reward prediction, and the distance between the prior and posterior distributions of state transitions until convergence.

[0020] Step 203: Using the model trained in step 202 as a new partially observable Markov decision process, use it to train the actor-commentator algorithm, minimize the loss between trajectory rewards and state values, and train the actor network based on the state value function predicted by the commentator.

[0021] The specific implementation process of the reinforcement learning online update module deployed to real-world scenarios is as follows:

[0022] Step 300: Deploy the system to a real-world scenario, model the user data in the real-world scenario into an observation sequence, use the encoder combined with the cyclic state space model to obtain the hidden state, and use it as the input of the actor to obtain the optimal action, i.e. the current recommended product;

[0023] Step 301: The new user data collected after system deployment is stored as a new trajectory in the experience playback.

[0024] Step 302: Periodically sample data from the experience replay to update the encoder, decoder, recurrent state space model, and actor and commentator networks; then repeat step 301.

[0025] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal product recommendation system based on contrastive learning and model-based reinforcement learning as described above.

[0026] A computer-readable storage medium storing a multimodal product recommendation system that performs the above-described contrastive learning and model-based reinforcement learning.

[0027] Beneficial Effects: Compared with existing technologies, the multimodal product recommendation system based on contrastive learning and model-based reinforcement learning provided by this invention uses various contrastive learning methods to learn the multimodal fusion representation of products, targeting their current multimodal attributes. By using the fusion features of other products with the same label as positive samples and the fusion features of products with different labels as negative samples, it fully considers the inherent correlation between products with the same label, thus solving the problem of difficult modal fusion. This invention also considers the sparsity of individual user behavior data, making full use of behavior data from different users. Based on past recommended products and behavior data, it constructs a partially observable Markov decision process, training the recommendation strategy in new environments, making it more robust. Compared with traditional methods, this method can be updated online and can be directly used for new users, solving the cold start problem. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the implementation of the multimodal feature fusion module based on contrastive learning in this embodiment of the invention.

[0029] Figure 2 This is a flowchart illustrating the implementation of the offline pre-training module of the recommendation system based on model reinforcement learning in this embodiment of the invention.

[0030] Figure 3 This is a flowchart illustrating the implementation of the online update module for reinforcement learning deployed in a real-world scenario in an embodiment of the present invention. Implementation

[0031] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0032] A multimodal product recommendation system based on contrastive learning and model-based reinforcement learning comprises three parts: a multimodal feature fusion module based on contrastive learning, an offline pre-training module for the recommendation system based on model-based reinforcement learning, and an online update module for deploying the system to real-world scenarios using reinforcement learning.

[0033] The implementation process of the multimodal feature fusion module based on contrastive learning is as follows: Figure 1As shown. First, obtain the product images, text, and tag information from the platform (step 10); then, starting with image feature construction, build an image feature extraction model (step 11-1); perform data augmentation operations such as inversion on the image data, using it as positive samples and other product images as negative samples, and use contrastive learning to estimate the loss through information noise to perform gradient descent and parameter updates on the model (step 11-2); repeat steps 11-1 and 11-2 until the loss curve converges (step 11-3); then, starting with text feature construction, build a text feature extraction model (step 12-1); perform data augmentation operations such as translation and re-translation on the text data, using it as positive samples and other product images as negative samples, and use contrastive learning to estimate the loss through information noise to perform gradient descent and parameter updates on the model (step 11-2); repeat steps 11-1 and 11-2 until the loss curve converges (step 11-3); then, starting with text feature construction, build a text feature extraction model (step 12-1); perform data augmentation operations such as translation and re-translation on the text data, using it as positive samples and other product images as negative samples. He uses product images as negative samples, employs contrastive learning to estimate the loss using information noise contrast, and trains the model using a masked language model, performing gradient descent and parameter updates (step 12-2); repeats steps 12-1 and 12-2 until the loss curve converges (step 12-3); finally, he constructs a product feature extraction model, using product information from the models in steps 11-1 and 12-1, then samples from the labels, using product images and text representations with the same label as positive samples and different labels as negative samples, using information noise contrastive estimation to estimate the loss, and performing gradient descent and parameter updates (step 13-2) until convergence (step 13-3); outputs the model (step 14).

[0034] The implementation process of the offline pre-training module of the recommendation system based on model reinforcement learning is as follows: Figure 2As shown. First, for the collected user data, including the recommended products, a multimodal feature fusion module based on contrastive learning is used to obtain the fused features of product images and text modalities (step 20). Next, the currently recommended products and the user's actions are used as observations in a partially observable Markov model. The sequence of recommended products and the sequence of actions in a single interaction trajectory are used as states, the action is the product recommended to the user by the system, the transition model is to add the new observation to the past state as the new state, and the reward function is to give positive feedback if the user buys or browses an item, and negative feedback if no browsing, all of which are numerical (step 21). Samples are taken from the generated trajectory data, the observations are input into the encoder, and the results and actions are input into the recurrent state space model to generate hidden states, constructing... Build a state transition model and a reward model, and reconstruct the observations from the hidden states using a decoder (step 22-1); based on the reconstruction observation loss, the reward model prediction loss, and the transition model distribution loss, perform gradient descent and parameter updates on the model (step 22-2); repeat training until the model converges (step 22-3); then, create two models, an actor and a critic (step 23-1); in the constructed reinforcement learning model, continuously interact with it to collect hidden state trajectory data, use the state reward to update the state value function, use the state value function to update the actor network, and perform gradient descent and parameter updates on the model (step 23-2); repeat training until the model converges (step 23-3); output the reinforcement learning environment model and the actor-critic model (step 24).

[0035] The implementation process of the reinforcement learning online update module deployed to real-world scenarios is as follows: Figure 3 As shown. First, the system trained by the offline pre-training module of the recommendation system based on model reinforcement learning is deployed to a real scene (step 30); for newly generated user interactions in the real scene, reinforcement learning observations and states are constructed according to step 21 in the offline pre-training module of the recommendation system based on model reinforcement learning (step 31); user behavior observations are input into the reinforcement learning model, and the recommended actions to users are obtained through actors (step 32); the collected user data is stored in the experience replay pool (step 33); data is periodically sampled from the experience replay pool (step 34); using this data, the reinforcement learning environment model and actor-commentator model are updated according to step 23-2 in the offline pre-training module of the recommendation system based on model reinforcement learning (step 35); if new user behaviors are generated, the above steps are repeated (step 36); if not, the process ends (step 37).

[0036] Obviously, those skilled in the art should understand that the modules of the multimodal product recommendation system based on contrastive learning and model-based reinforcement learning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

Claims

1. A multimodal product recommendation system based on contrastive learning and model-based reinforcement learning, characterized in that, It includes a multimodal feature fusion module based on contrastive learning, an offline pre-training module for a recommendation system based on model-based reinforcement learning, and an online update module for the system to be deployed to real-world scenarios using reinforcement learning. In the multimodal feature fusion module based on contrastive learning: information about recommended products is collected, including images, text descriptions, and tags; data augmentation is performed on the images, and contrastive learning is used to learn image representations; text representations of product descriptions are learned using contrastive learning methods; and multimodal representations of products with the same tags are learned using contrastive learning methods by combining image and text representations. In the offline pre-training module of the recommendation system based on model-based reinforcement learning: the observations, states, actions, transition functions, and reward functions of partially observable reinforcement learning are defined; the collected user behavior data and product data are used as offline reinforcement learning trajectory data, the observations are generated into embeddings through the encoder, the hidden states are constructed through the embeddings, and the recurrent state space model is trained; the trained model is used to learn the policy using the actor-commentator method. The system is deployed in a real-world reinforcement learning online update module: For users in real-world scenarios, their data is encoded into a recurrent state space model, and product recommendations are given to users based on the results of the actors; using continuously acquired user behavior data, the data is used to construct the trajectory of the reinforcement learning model, thereby updating the recurrent state space model, the actor neural network, and the commentator neural network.

2. The multimodal product recommendation system based on contrastive learning and model-based reinforcement learning according to claim 1, characterized in that, The specific implementation process of the multimodal feature fusion module based on contrastive learning is as follows: Step 100: Extract product images, text descriptions, and tags from the platform to form a dataset; Step 101: Construct an image feature extraction neural network; for each image, perform data augmentation, and train the network using the original image, the augmented image, and other images, employing information noise contrast estimation loss until convergence. Step 102: Construct a text feature extraction neural network; For the text description of the product, data augmentation is performed using the original text, the data-augmented text, and other text, and the information noise contrast estimation loss is used for training; for the text description of the product, a masked language model is used for training. Step 103: Construct a neural network that fuses image and text features; using the image and text features extracted from the products in steps 101 and 102, input them into the fusion feature network, use the fusion features of other products with the same label as positive samples, and the fusion features of products with different labels as negative samples, and use information noise contrast estimation loss to train until convergence.

3. The multimodal product recommendation system based on contrastive learning and model-based reinforcement learning according to claim 2, characterized in that, The specific implementation process of the offline pre-training module of the recommendation system based on model reinforcement learning is as follows: Step 200: Define state S_T as the recommended products and user behavior data from time 1 to the current time T. The recommended products are represented by the data obtained in step 103. The observations are the recommended products and user behavior responses at the current time. The actions are the products recommended by the system to the user. The transition function is the new recommended product and user behavior sequence S_{T+1} reached after the current recommended action and the user's behavior response in the previous state S_T. The reward function is the numerical response of the user to the currently recommended product. Construct the above offline trajectory sequence from the existing user behavior data. Step 201: Construct the encoder and decoder; construct a cyclic state-space model for modeling environmental changes; Step 202: Use the offline trajectory sequence obtained in step 200 to train the encoder and decoder in step 201, as well as the cyclic state space model, to minimize the reconstruction loss, reward prediction, and the distance between the prior and posterior distributions of state transitions until convergence. Step 203: Using the model trained in step 202 as a new partially observable Markov decision process, use it to train the actor-commentator algorithm, minimize the loss between trajectory rewards and state values, and train the actor network based on the state value function predicted by the commentator.

4. The multimodal product recommendation system based on contrastive learning and model-based reinforcement learning according to claim 1, characterized in that, The specific implementation process of the reinforcement learning online update module deployed to real-world scenarios is as follows: Step 300: Deploy the system to a real-world scenario, model the user data in the real-world scenario into an observation sequence, use the encoder combined with the cyclic state space model to obtain the hidden state, and use it as the input of the actor to obtain the optimal action, i.e. the current recommended product; Step 301: The new user data collected after system deployment is stored as a new trajectory in the experience playback. Step 302: Periodically sample data from the experience replay to update the encoder, decoder, recurrent state space model, and actor and commentator networks; then repeat step 301.

5. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a multimodal product recommendation system based on contrastive learning and model-based reinforcement learning as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal product recommendation system that performs any one of the claims 1-4 based on contrastive learning and model-based reinforcement learning.

Images