An offline reinforcement learning commodity recommendation system based on multi-modal contrast learning

Abstract

The application discloses an offline reinforcement learning commodity recommendation system based on a multi-modal contrast learning algorithm. First, the related images of commodities, the texts describing the commodities and the related features of other commodities are acquired, the information of multiple modes is fused through a contrast learning algorithm, and the feature representation of the commodities is obtained. The commodity recommendation process is modeled as a reinforcement learning problem, the state of a user is obtained through the commodity features of the commodities on which the user has ever generated behaviors and the basic information of the user, the commodities recommended by the system are taken as actions, and the feedback of the user for the recommended commodities is taken as rewards. The offline reinforcement learning algorithm is used to train on the existing user behavior log data, and the strategy of the recommendation system is obtained. The application uses the contrast learning algorithm to comprehensively integrate the information of multiple modes to construct the features of the commodities, under the premise of modeling the recommendation system as a reinforcement learning problem, the recommendation strategy is obtained through the offline reinforcement learning algorithm, so that the commodity recommendation system can better complete the recommendation task.

Description

TECHNICAL FIELD

[0001] The present application relates to an offline reinforcement learning commodity recommendation system based on multi-modal contrast learning, relates to multi-modal information acquisition and processing, and combines machine learning to improve the performance of the commodity recommendation system. BACKGROUND

[0002] In the current commodity recommendation system, the user behavior data required by reinforcement learning must be obtained from the actual use scene, and the online reinforcement learning cycle is long and it is difficult to provide stable recommendation services for users during the learning process, which is costly. In addition, how to embed more rich information in the user data required by reinforcement learning is also a big problem. The existing reinforcement learning data often concerns the behavior of users on commodities, and the attributes of commodities are difficult to combine into the data of reinforcement learning. The insufficient information of the data required by reinforcement learning greatly affects the commodity recommendation effect of reinforcement learning. In summary, the application of reinforcement learning in the commodity recommendation system has the problems of instability, poor effect, etc., which makes it difficult to apply in practice. SUMMARY

[0003] The present application provides an offline reinforcement learning commodity recommendation system based on multi-modal contrast learning, which embeds commodity attributes into the data of reinforcement learning through multi-modal contrast learning, and adopts the method of offline reinforcement learning, which alleviates the instability, poor effect, etc. of reinforcement learning in the commodity recommendation system, and can provide stable commodity recommendation services for users. The features of multiple modal information are integrated by contrast learning, and the user state is composed of these features. The offline reinforcement learning algorithm is used to generate the recommendation strategy of the commodity, and finally a more accurate and stable commodity recommendation service is provided for the user, and the performance of the commodity recommendation system is improved.

[0004] Technical scheme: An offline reinforcement learning commodity recommendation system based on multi-modal contrast learning, comprising a multi-modal information acquisition module, a model construction module and a commodity information recommendation module.

[0005] The multi-modal information acquisition module uses contrast learning to integrate the feature representation of commodity multi-modal information;

[0006] Collect the multi-modal information of the image of the commodity, the text describing the commodity and the basic characteristics of the commodity; use the contrast learning algorithm to integrate the multi-modal information to form the feature representation of the commodity, denoted as E;

[0007] The model construction module models the commodity recommendation system as a reinforcement learning problem;

[0008] The feature representation of the commodity in which the user generates the behavior and the basic information of the user (for example, age, gender, region, etc.) constitute the current moment state of the user, denoted as S_t; the commodity recommended by the recommendation system for the user state is denoted as a; the feedback of the user to the recommended commodity is denoted as r; the next moment state of the user is composed of the commodity in which the user generates the behavior, the commodity in which the user generates the behavior before and the basic information of the user, denoted as S_t+1;

[0009] The commodity information recommendation module generates a commodity information recommendation strategy by using an offline reinforcement learning algorithm.

[0010] First, the user behavior log is sorted according to time; then, the commodity features are obtained from the commodity in which the user generates the behavior in the log, and the commodity features and the basic information of the user are combined as S_t; the next commodity in which the user generates the behavior in the log and a randomly selected commodity are selected as the recommended commodity given by the recommendation system as a; the feedback of the user to the recommended commodity is taken as the reward r, and the behavior of the user after the recommended commodity is added to the log, and the new commodity features generated by the behavior and S_t constitute the next moment state S_t+1 of the user; St, a, r, S_t+1 jointly constitute a transition tuple, and multiple transition tuples constitute an offline data set, and a recommendation strategy of the commodity recommendation system is trained by using an offline reinforcement learning algorithm on the offline data set.

[0011] In an online environment, the trained strategy is used to recommend commodity information for the user in real time.

[0012] The implementation process of the multi-modal information acquisition module for comprehensively acquiring feature representations of multi-modal information of commodities is as follows:

[0013] Step 100, collect multi-modal information of commodity images, text describing commodities and basic features of commodities;

[0014] Step 101, translate the text information describing the commodity into another language through a translation tool, and then translate the another language back to the original language to obtain a text information with similar meaning but different expression, and the text information and the original commodity image and commodity basic features constitute a positive sample;

[0015] Step 102, text feature extraction is performed on the text information describing the commodity to obtain TF-IDF feature representation, and similar text information describing commodities is searched in a commodity text library according to the TF-IDF feature representation, and the similar text information describing commodities and the original commodity image and commodity basic features constitute a negative sample;

[0016] Step 103, sort the basic features of the commodity according to the number of non-zero values, starting from the sparsest feature, and sequentially calculating the proportion of non-zero values with the following features, if the degree of mutual exclusion is lower than the threshold, the two features are added to a mutual exclusion feature set at the same time, repeat the above process until all features are added to the mutual exclusion feature set; for the features in the same mutual exclusion feature set, one of the original basic features of the commodity can be changed to the feature in the same mutual exclusion feature set, and the changed basic features of the commodity and the original commodity image and the text describing the commodity constitute the negative sample;

[0017] Step 104, according to the positive sample constructed in step 101, and the original commodity sample to form a positive sample pair, according to the negative sample pair constructed in step 102 and step 103, and the original commodity sample to form a negative sample pair, apply the contrast learning loss to maximize the similarity between different modal features of the same commodity sample, and minimize the similarity between different commodity sample features, and obtain the feature representation of the commodity.

[0018] The implementation process of modeling the commodity recommendation system as a reinforcement learning problem by the model construction module is specifically:

[0019] Step 200, collect user logs, including user basic information and user behavior record information of commodities;

[0020] Step 201, extract user basic information and user behavior generated goods from user logs, and obtain the feature representation of the goods, the state S_t of the user at the current time can be obtained through the user's basic information and the feature representation of the goods before a certain time t;

[0021] Step 202, find the goods in which the user generates behavior in the next time in the user log, and randomly select goods from all goods to jointly constitute the commodity recommendation list generated by the recommendation system for the user, which can be considered as the action a executed by the commodity recommendation system;

[0022] Step 203, according to the intersection of the goods in which the user generates behavior after the current time and the goods in the recommendation list, the satisfaction degree of the user to the recommendation list can be obtained, and the ratio of the number of intersection elements to the number of goods in the recommendation list is regarded as the reward r;

[0023] Step 204, after the user interacts with the commodity recommendation list, according to the goods in which the user generates behavior before the next time and the basic information of the user, the state S_t+1 of the user in the next time is constituted.

[0024] The process of generating a commodity recommendation strategy by the commodity information recommendation module using an offline reinforcement learning algorithm is specifically:

[0025] Step 300, using the <S_t, a, r, S_t+1> obtained in steps 201, 202, 203 and 204 to form a transition tuple, repeating steps 201, 202, 203 and 204 to build an offline data set composed of multiple transition tuples;

[0026] Step 301, training the recommendation system on the offline data set using an offline reinforcement learning algorithm to optimize the expected return and obtain a commodity recommendation strategy.

[0027] The offline reinforcement learning algorithm can be any algorithm in the field of offline reinforcement learning. Using such an algorithm, we can generate a good commodity recommendation strategy through the offline data set without interacting with the user.

[0028] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above-mentioned computer program to realize the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning as described above.

[0029] A computer readable storage medium stores a computer program for executing the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning as described above.

[0030] Advantages: Compared with the prior art, the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning provided by the present application can generate features by comprehensively considering multi-modal information of commodities, and generate appropriate commodity recommendation strategies in an offline environment, thereby providing stable and good commodity recommendation information for users. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A training flowchart for obtaining feature representations by comprehensively considering multi-modal information using contrast learning in the present application;

[0032] Figure 2 A flowchart for generating commodity recommendation strategies using offline reinforcement learning algorithms in the present application. DETAILED DESCRIPTION

[0033] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrating the present application and not limiting the scope of the present application. After reading the present application, those skilled in the art can make various modifications to the present application, and such modifications fall within the scope defined by the claims attached hereto.

[0034] The offline reinforcement learning commodity recommendation system based on multi-modal contrast learning includes a multi-modal information acquisition module, a model construction module and a commodity information recommendation module.

[0035] The multi-modal information acquisition module utilizes contrast learning to comprehensively acquire multi-modal information of the commodity to obtain a feature representation of the commodity.

[0036] The model construction module models the commodity recommendation system as a reinforcement learning problem.

[0037] The commodity information recommendation module utilizes an offline reinforcement learning algorithm to generate a commodity information recommendation strategy.

[0038] In an online environment, the trained strategy is utilized to perform real-time commodity information recommendation for users.

[0039] The process of the multi-modal information acquisition module comprehensively acquiring multi-modal information to obtain a feature of the commodity through contrast learning is shown in Figure 1 . First, multi-modal data about the commodity is acquired from the mall, such as text, images, and basic features (step 10); then, the text is translated multiple times, for example, translated into another language and then translated back to the original language to obtain positive samples about the text (step 11); the TF-IDF features of the text samples are calculated, and similar texts with different semantics are obtained according to the feature similarity as negative samples (step 12); a mutual exclusive feature set is constructed for the basic features of the commodity, and the basic features of the commodity are sorted according to the number of non-zero values, starting from the most sparse feature, and sequentially calculating the proportion of non-zero values with the following features, if the mutual exclusive degree is lower than the threshold, both features are added to a mutual exclusive feature set, and the mutual exclusive feature set construction process is repeated until all features are added to the mutual exclusive feature set; the features in the mutual exclusive feature set to which the original features of the commodity belong are used to replace the original feature attributes of the commodity to obtain negative samples about the basic features of the commodity (step 13); the original commodity multi-modal information and the positive samples obtained above form a positive sample pair, and the negative samples form a negative sample pair, the contrast loss is calculated, and gradient descent is performed to learn the feature representation of the commodity; steps 10, 11, 12, and 13 are repeated until the training converges or ends.

[0040] The process of utilizing an offline reinforcement learning algorithm to generate a commodity recommendation strategy is shown in Figure 2 . First, the feature representation of the commodity is obtained through the training process shown in Figure 1 , the model construction module models the commodity recommendation system as a reinforcement learning problem, obtains the representation of the current state of the user, the action of the recommendation system, the reward, and the next state of the user, and constitutes an offline data set.

[0041] The implementation process of the model construction module modeling the commodity recommendation system as a reinforcement learning problem is as follows:

[0042] Step 200, collect user logs, including basic information of the user and behavior record information of the user on the commodity;

[0043] Step 201, extracting user basic information and user generated behavior goods from the user log, and obtaining the feature representation of the goods, the state S_t of the user at the current time can be obtained through the basic information of the user and the feature representation of the goods before a certain time t;

[0044] Step 202, find the goods generated by the user in the next time in the user log, and randomly select goods in all goods to form the goods recommendation list generated by the recommendation system for the user, which can be considered as the action a executed by the goods recommendation system;

[0045] Step 203, according to the intersection of the goods generated by the user after the current time and the goods in the recommendation list, the satisfaction degree of the user to the recommendation list can be obtained, and the ratio of the number of intersection elements to the number of goods in the recommendation list is regarded as the reward r;

[0046] Step 204, after the user interacts with the goods recommendation list, according to the goods generated by the user before the next time and the basic information of the user, the state S_t+1 of the user at the next time is constructed.

[0047] The commodity recommendation system obtains the transition tuple of the user from the offline data set (step 30); the commodity recommendation system trains the commodity recommendation strategy through the offline reinforcement learning algorithm (step 31); it is judged whether the training converges, yes, the training is ended, and the obtained commodity recommendation strategy can be used for the commodity recommendation system; no, the transition tuple is obtained from the offline data set to continue training.

[0048] Obviously, those skilled in the art should understand that the modules of the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning of the embodiments of the application described above can be realized by general computing devices, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described here can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the embodiments of the application are not limited to any specific combination of hardware and software.

Claims

1. An offline reinforcement learning item recommendation system based on multi-modal contrastive learning, characterized in that, The system comprises a multi-modal information acquisition module, a model construction module and a commodity information recommendation module. The multi-modal information acquisition module uses contrast learning to comprehensively acquire feature representation of commodity multi-modal information. The model construction module models the commodity recommendation system as a reinforcement learning problem. The commodity information recommendation module uses an offline reinforcement learning algorithm to generate a commodity information recommendation strategy. The implementation process of the multi-modal information acquisition module using contrast learning to comprehensively acquire feature representation of commodity multi-modal information is as follows: Step 100: Collect multi-modal information of commodity images, text describing commodities and commodity basic features. Step 101: Translate the text information describing commodities into another language through a translation tool, and then translate the another language back to the original language to obtain a text information with similar meaning but different expression, which together with the original commodity image and commodity basic features constitutes a positive sample. Step 102: Extract text features from the text information describing commodities to obtain TF-IDF feature representation, and search for similar text information describing commodities in a commodity text library according to the TF-IDF feature representation, which together with the original commodity image and commodity basic features constitutes a negative sample. Step 103: Sort the commodity basic features according to the number of non-zero values, and start from the most sparse feature to calculate the proportion of non-zero values with the following features, if the mutual exclusion degree is lower than a threshold, add both features to a mutual exclusion feature set, repeat the step until all features are added to the mutual exclusion feature set; for the features in the same mutual exclusion feature set, change one feature in the original commodity basic features to the feature in the same mutual exclusion feature set, and the changed commodity basic features and the original commodity image and text describing commodities constitute a negative sample. Step 104: According to the positive sample constructed in step 101, form a positive sample pair with the original commodity sample, and according to the negative sample constructed in steps 102 and 103, form a negative sample pair with the original commodity sample, apply contrast learning loss to maximize the similarity between different modal features of the same commodity sample and minimize the similarity between different commodity sample features, and obtain the feature representation of the commodity. The implementation process of the model construction module modeling the commodity recommendation system as a reinforcement learning problem is as follows: Step 200: Collect user logs, including user basic information and user behavior record information of commodities; Step 201: Extract user basic information and user behavior generating commodities from user logs, and obtain feature representation of commodities, and get the state S_t of the user at the current time through the user basic information and the feature representation of the commodities before time t; Step 202: Find the commodities generating behaviors of the user at the next time in the user logs, and randomly select commodities in all commodities to jointly constitute the commodity recommendation list generated by the recommendation system for the user, which is considered as the action a executed by the commodity recommendation system. In step 203, the satisfaction degree of the user to the recommendation list can be obtained according to the intersection of the generated behavior of the user after the current time and the commodities in the recommendation list, and the ratio of the number of intersection elements to the number of commodities in the recommendation list is regarded as the reward r. In step 204, after the user interacts with the commodity recommendation list, the state S_t+1 of the user at the next time is formed according to the commodities on which the user generates behavior and the basic information of the user before the next time.

2. The multi-modal contrastive learning based offline reinforcement learning item recommendation system of claim 1, wherein, The multi-modal information acquisition module collects multi-modal information of images of the commodities, texts describing the commodities and basic features of the commodities; and uses a contrast learning algorithm to integrate the multi-modal information to form a feature representation of the commodities, denoted as E.

3. The multi-modal contrastive learning based offline reinforcement learning item recommendation system of claim 1, wherein, The model construction module forms the current time state of the user by using the feature representation of the commodities on which the user generates behavior and the basic information of the user, denoted as S_t. The commodities recommended by the recommendation system for the user state are denoted as a, the feedback of the user to the recommended commodities is denoted as r, and the next time state of the user is formed by using the commodities on which the user generates behavior, the commodities on which the user generates behavior before the user and the basic information of the user, denoted as S_t+1.

4. The multi-modal contrastive learning based offline reinforcement learning item recommendation system of claim 1, wherein, The commodity information recommendation module generates a commodity information recommendation strategy by using an offline reinforcement learning algorithm. First, the user behavior log is sorted according to time; then, the commodity features are obtained by using the commodities on which the user generates behavior in the log, and the commodity features and the basic information of the user are combined as S_t; the next commodity on which the user generates behavior in the log and a randomly selected commodity are selected as the recommended commodities given by the recommendation system as a; the feedback of the user to the recommended commodities is regarded as the reward r, the behavior of the user after the recommended commodities is added to the log, and the new commodity features on which the user generates behavior and S_t form the next time state S_t+1 of the user; St, a, r and S_t+1 jointly form a transition tuple, and multiple transition tuples form an offline data set, and the recommendation strategy of the commodity recommendation system is generated by using an offline reinforcement learning algorithm on the offline data set.

5. The multi-modal contrastive learning based offline reinforcement learning item recommendation system of claim 1, wherein, The process of generating the commodity recommendation strategy by using the offline reinforcement learning algorithm by the commodity information recommendation module is specifically as follows: In step 300, the transition tuples are formed by using <S_t, a, r, S_t+1> obtained in steps 201, 202, 203 and 204, and the steps 201, 202, 203 and 204 are repeated to construct multiple transition tuples to form an offline data set; In step 301, the recommendation system is trained on the offline data set by using an offline reinforcement learning algorithm to optimize the expected return, and the commodity recommendation strategy is obtained.

6. A computer device, characterized by: The computer device includes a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning according to any one of claims 1-5 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program for executing the offline reinforcement learning commodity recommendation system based on multi-modal contrast learning according to any one of claims 1-5.

Images