A user personalized recommendation method based on multimodal data

Abstract

The present invention provides a user personalized recommendation method based on multimodal data, and relates to the field of network recommendation technology. After obtaining the historical behavior records and image logs that the user allows to be collected, the method extracts all the objects involved therein, as well as the features of each object to be recommended, and maps them to the same multidimensional space, so as to integrate the multimodal data, and then uses the reinforcement learning model as the recommendation system agent, trains the agent through the collected user records, and uses the trained recommendation agent to make recommendations, thereby realizing user personalized recommendations for multimodal data. The user historical behavior records, image logs, and the set of objects to be recommended in the method proposed by the present invention can all contain objects in multiple fields (such as text, pictures, videos, etc.), and blur the distinction between multimodalities through the method of post-feature extraction integration, thereby solving the problem that a traditional recommendation system can only be applied to a single field for recommendation.

Description

Technical Field

[0001] The present invention relates to the technical field of network recommendation, and in particular to a user personalized recommendation method based on multimodal data. Background Art

[0002] In recent years, with the rapid development of the Internet, modern society has become an information-based and digital society. Data is flooding the world, and information explosion has become the norm. However, faced with a large amount of data, users' utilization of information has decreased, which has led to the problem of information overload. In this regard, the recommendation system is one of the key technologies to effectively solve the problem of information overload. In fact, with the rapid development of the Internet, the Internet of Things, and cloud computing technologies, personalized recommendation systems have become standard for Internet products. Internet users in e-commerce, video, news, music and other fields will face Internet information that is closely related to the recommendation system.

[0003] The recommendation system obtains the user's historical behavior data, such as web browsing data, purchase history or project ratings, and then makes personalized recommendations to users. Solutions to the recommendation problem include content filtering-based recommendation systems, collaborative filtering-based recommendation systems, and hybrid recommendation systems. With the rise of deep learning algorithms, various recommendation algorithms using neural networks are also widely used in the implementation of recommendation systems, such as recursive neural networks, convolutional neural networks, generative adversarial networks, etc. In addition, there are also recommendation technologies based on reinforcement learning and heterogeneous networks.

[0004] Most current recommendation systems only provide solutions to the problem of user personalized recommendation based on a single modality, that is, they only learn users' historical data in a single field (such as news, video, music, etc.). Therefore, they can only capture users' interest preferences in a single field and make recommendations based on them, which greatly limits the diversity of recommendation results. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a user personalized recommendation method based on multimodal data. According to the historical data formed by users in various fields (such as news, video, music, etc.), user preferences are learned and user portraits are formed, so that multimodal data can be recommended to expand the diversity of recommendation results.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A user personalized recommendation method based on multimodal data comprises the following steps:

[0008] Step 1: Obtain the total object set I and the object set to be recommended L0, and clarify the types of multimodal information that need to be addressed.

[0009] Step 2: For each modality, apply the corresponding feature extraction algorithm to extract features from each object in the total object set and map them into the same mathematical multidimensional space S.

[0010] Step 3: Obtain and accumulate user historical behavior records and image logs. The historical behavior record is the user's click history before the image, that is, the list of objects that have been clicked; the image log is the list of objects displayed to the user and the user's click behavior on these objects, 1 means click, and 0 means no click.

[0011] Step 4: Initialize the recommendation system, input the multidimensional space S obtained in step 2 and the user's historical behavior records and image logs obtained in step 3 into the recommendation system, and set the recommendation system agent and reinforcement learning environment parameters.

[0012] Step 5: Perform training of the recommender agent.

[0013] Step 6: Use the trained recommendation system agent to process the set of objects to be recommended L0, use the recommendation system to simulate the user's interactive behavior towards the set of objects to be recommended L0, and generate the corresponding image log D.

[0014] Step 7: Extract the set of objects that the user interacted with in the image log D, that is, the set of personalized interaction objects predicted by the recommendation system agent for the user, as the total recommendation list L.

[0015] Step 8: Process the total recommendation list L according to different requirements and generate a multimodal recommendation list.

[0016] Step 9: Recommend the generated multimodal recommendation list to obtain specific user interaction results, thereby generating an image log. Whenever the logs accumulate to a given n, further train and update the agent.

[0017] The beneficial effect of adopting the above technical solution is that: the user personalized recommendation method based on multimodal data provided by the present invention, after obtaining the historical behavior records and image logs that the user allows to collect, extracts all the objects involved therein, as well as the features of each object to be recommended, and maps them to the same multidimensional space, so as to integrate the multimodal data, and then uses the reinforcement learning model as the recommendation system agent, trains the agent through the collected user records, and uses the trained recommendation agent to make recommendations, thereby realizing user personalized recommendations for multimodal data. Compared with the prior art, the user historical behavior records, image logs, and the set of objects to be recommended in the method proposed by the present invention can all contain objects in multiple fields (such as text, pictures, videos, etc.), and blur the distinction between multimodalities through the method of post-feature extraction integration, thereby solving the problem that a traditional recommendation system can only be applied to a single field for recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a recommended method provided by an embodiment of the present invention;

[0019] Figure 2 A flow chart of a training method for an intelligent agent (rainbow model) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0021] like Figure 1 As shown, the user personalized recommendation method based on multimodal data in this embodiment is described as follows.

[0022] Step 1: Obtain the total object set I and the object set to be recommended L0, and clarify the types of multimodal information that need to be addressed.

[0023] Step 2: For each modality, apply the corresponding feature extraction algorithm to extract features from each object in the total object set and map them to the same mathematical multidimensional space S. For example, images use CNN or RNN convolutional neural network to extract features, text uses tf-idf algorithm or cnn convolutional neural network to extract features, videos use P3D (Pseudo-3DResidual Networks) residual network to extract features, audio uses discrete wavelet transform (DWT) or perceptual linear prediction (PLP) algorithm to extract features, etc.

[0024] Step 3: Obtain and accumulate user historical behavior records and image logs. The historical behavior record is the user's click history before the image, that is, the list of objects that have been clicked; the image log is the list of objects displayed to the user and the user's click behavior on these objects, 1 means click, and 0 means no click.

[0025] Step 4: Initialize the recommendation system, input the multidimensional space S obtained in step 2 and the user's historical behavior records and image logs obtained in step 3 into the recommendation system, and set the recommendation system agent and reinforcement learning environment parameters.

[0026] Step 5: Perform training of the recommender system agent, such as Figure 2 As shown, the process is as follows:

[0027] Step 5.1: The agent requests a user-related record from the user's historical behavior records and image logs obtained in step 3.

[0028] Step 5.2: The memory returns the record and replaces the corresponding object with the feature representation through the result of step 2.

[0029] Step 5.3: The agent forms a user profile based on the user's historical behavior records, and executes the image in sequence according to the user profile. If the agent's actions are consistent with the real log, the agent will receive a reward. If the agent's actions are inconsistent with the real log, no reward will be received or a penalty will be received.

[0030] Step 5.4: After executing all the entries in the user image log, it is considered that a reinforcement learning frame is completed. Judgment: If the training termination condition in the reinforcement learning environment parameters set in step 4 is met, the agent training is completed and go to step 6, otherwise return to step 5.1.

[0031] Step 6: Use the trained recommendation system agent to process the set of objects to be recommended L0, use the recommendation system to simulate the user's interactive behavior towards the set of objects to be recommended L0, and generate the corresponding image log D.

[0032] Step 7: Extract the set of objects that the user interacted with in the image log D, that is, the set of personalized interaction objects predicted by the recommendation system agent for the user, as the total recommendation list L.

[0033] Step 8: Process the total recommendation list L according to different requirements and generate a multimodal recommendation list;

[0034] When it is necessary to explore the user's broad interests in multiple fields, or when diversified recommendation result modalities are required, the total recommendation list L can be simply sorted according to the specific scenario requirements and then output to generate a multimodal recommendation list L1.

[0035] When it is necessary to make recommendations in other fields to users based on a certain object (such as providing pictures for news, recommending text hyperlinks for movies, etc.), other modal objects closest to the current object can be selected from the multidimensional space S according to the nearest neighbor principle to generate a multimodal recommendation list L2.

[0036] When it is necessary to combine knowledge expressions at different levels to make more accurate and effective recommendations to users, a multimodal fusion algorithm is used to fuse the features of different modalities, and the intelligent agent is trained based on the fused features to generate a recommendation list L3 with multimodal expressions for the same object, such as a news cover on a news title.

[0037] Step 9: Recommend the generated multimodal recommendation list to obtain specific user interaction results, thereby generating an image log. Whenever the logs accumulate to a given n, further train and update the agent.

[0038] This embodiment performs the method of the present invention in sequence based on the MIcrosoft News Dataset (abbreviated as MIND):

[0039] Step 1: Since the MIND dataset only contains news text data, it does not meet the requirements of multimodal tasks. Therefore, according to the news links provided in the MIND dataset, the spiderFlow crawler tool (https: / / github.com / chenyuansgit / spiderFlow) is used to crawl the corresponding image data contained in the news to form an image dataset corresponding to the news text. Therefore, the total object set I obtained is all news texts and news pictures, and the recommended object set L0 is the image log news text and the corresponding news picture set in the test set. It is clear that the multimodal information types that need to be faced are text and images.

[0040] Step 2: For the two modalities of text and image, the corresponding feature extraction algorithms are applied respectively. The news text features are extracted by weighting the news text features through the Tf-idf term, and then feature selection is performed through the chi-square test algorithm to reduce the dimensions of the basic text features obtained above, so that the model has stronger generalization ability, reduces overfitting, and enhances the understanding of features and feature values. The news image features are extracted through the pre-trained VGG16 neural network in the keras library, and finally the extracted features of each object in the total object set are mapped to the same multidimensional space S in mathematics through the umap algorithm (https: / / arxiv.org / abs / 1802.03426).

[0041] Step 3: Read the user historical behavior records and image logs accumulated in the dataset.

[0042] Step 4: Initialize the recommendation system, input the multidimensional space S obtained in step 2 and the user's historical behavior records and image logs obtained in step 3 into the recommendation system, and set the recommendation system agent and reinforcement learning environment parameters (taking the rainbow model as an example): the action space is an integer from 0 to 10, and the probability of the object being clicked is predicted when it is divided by 10; the state space is a multidimensional matrix that reflects the user portrait and the characteristics of the current object to be determined; the reward is determined by the auc indicator obtained from the prediction result of the image log output.

[0043] Step 5: Perform training of the recommender system agent, such as Figure 2 As shown, the process is as follows:

[0044] Step 5.1: The agent requests a user-related record from the set obtained in step 3;

[0045] Step 5.2: The memory returns the record, and replaces the corresponding object with the feature representation through the result of step 2;

[0046] Step 5.3: The agent uses the Mean-Shift algorithm to cluster and separate user preference features based on the user's historical records to form a user portrait. It then predicts the user's click rate for the objects in the image log one by one based on the obtained user portrait, and rewards the agent based on the predicted results and the AUC indicator calculated from the actual records.

[0047] Step 5.4: After executing all the entries in the user image log, it is considered that a reinforcement learning frame is completed. Judgment: If the training termination condition in the reinforcement learning environment parameters set in step 4 is met, the agent training is completed, otherwise return to step 5.1.

[0048] Step 6: Use the trained recommendation system agent to process the image logs in the test set, predict the user's click probability for each object based on the user's historical behavior records in the test set, calculate the AUC index, and record the prediction result D.

[0049] Step 7: Use the objects with high click probability in the prediction result D as the recommended list L.

[0050] Step 8: According to different requirements, the total recommendation list L can be processed to generate a multimodal recommendation list; the following requirement processing process is as follows:

[0051] When it is necessary to explore the user's broad interests in multiple fields, or when diversified recommendation result modalities are required, the total recommendation list L can be simply sorted according to the specific scenario requirements and then output to generate a multimodal recommendation list L1.

[0052] When it is necessary to make recommendations in other fields to users based on a certain object (such as providing pictures for news, or recommending news hyperlinks for pictures, etc.), other modal objects that are closest to the current object can be selected from the multidimensional space S according to the nearest neighbor principle to generate a multimodal recommendation list L2.

[0053] Step 9: Recommend the generated multimodal recommendation list to obtain specific user interaction results, thereby generating an image log. Whenever the logs accumulate to a given n, the agent can be further trained and updated.

[0054] Through the above steps, the final experimental result indicators are shown in Table 1 (different recommendation models are used in step 5).

[0055] Table 1 Experimental results of various recommended models

[0056]

[0057] The reinforcement learning model described in the present invention is used as a recommendation system agent for recommendation. The method is tested on the MIcrosoft News Dataset (MIND for short). The results show that the performance indicators of the reinforcement learning agent and the traditional recommendation model are similar, that is, they are competent for recommendation tasks. The MIND dataset is expanded in the image mode by crawlers to make it a dataset of mixed modes of text and pictures, and further experiments are conducted. From the results of the experiments, the various indicators of the present invention can also meet the user's personalized recommendation needs for multimodal data, and meet the requirements of the recommendation system.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A user personalized recommendation method based on multimodal data, characterized by: The method comprises the following steps: Step 1: Obtain the total object set I and the object set to be recommended L0, and clarify the types of multimodal information that need to be targeted; Step 2: For each modality, apply the corresponding feature extraction algorithm to extract features from each object in the total object set and map them into the same multidimensional space S mathematically; Step 3: Obtain and accumulate user historical behavior records and image logs; Step 4: Initialize the recommendation system, input the multidimensional space S obtained in step 2 and the user's historical behavior records and image logs obtained in step 3 into the recommendation system, and set the recommendation system agent and reinforcement learning environment parameters; Step 5: Execute the training of the recommendation system agent; the specific process is: Step 5.1: The agent requests a user-related record from the user's historical behavior records and image logs obtained in step 3; Step 5.2: The memory returns the record, and replaces the corresponding object with the feature representation through the result of step 2; Step 5.3: The agent forms a user profile based on the user's historical behavior records, and executes the image in sequence according to the user profile. If the agent's actions are consistent with the real log, the agent will receive a reward. If the agent's actions are inconsistent with the real log, the agent will not receive a reward or will be punished. Step 5.4: After executing all the entries in the user image log, it is considered that a reinforcement learning frame is completed. If the training termination condition in the reinforcement learning environment parameters set in step 4 is met, the agent training is completed and go to step 6, otherwise return to step 5.1; Step 6: Use the trained recommendation system agent to process the set of objects to be recommended L0, use the recommendation system to simulate the user's interactive behavior towards the set of objects to be recommended L0, and generate the corresponding image log D; Step 7: Extract the set of objects that the user interacted with in the image log D, that is, the set of personalized interaction objects predicted by the recommendation system agent for the user, as the total recommendation list L; Step 8: Process the total recommendation list L according to different requirements and generate a multimodal recommendation list; Step 9: Recommend the generated multimodal recommendation list to obtain specific user interaction results, thereby generating an image log; whenever the logs accumulate to a given n, further train and update the agent.

2. The user personalized recommendation method based on multimodal data according to claim 1, characterized in that: The historical behavior record in step 3 is the user's click history before the image is formed, that is, the list of objects that have been clicked; the image log is the list of objects displayed to the user and the user's click behavior on these objects, 1 represents click, and 0 represents no click.

3. The user personalized recommendation method based on multimodal data according to claim 1 or 2, characterized in that: In step 2, features are extracted from images using a CNN convolutional neural network; features are extracted from text using a tf-idf algorithm or a CNN convolutional neural network; features are extracted from videos using a P3D residual network, and features are extracted from audio using a discrete wavelet transform or a perceptual linear prediction algorithm.

4. The user personalized recommendation method based on multimodal data according to claim 1 or 2, characterized in that: In step 8, when it is necessary to explore the user's broad interests in multiple fields, or to diversify the recommendation result modalities, the total recommendation list L is simply sorted according to the specific scenario requirements and then output to generate a multimodal recommendation list L1.

5. The user personalized recommendation method based on multimodal data according to claim 1 or 2, characterized in that: In step 8, when it is necessary to make recommendations in other fields to the user based on a certain object, other modal objects closest to the current object are selected from the multidimensional space S according to the nearest neighbor principle to generate a multimodal recommendation list L2.

6. The user personalized recommendation method based on multimodal data according to claim 1 or 2, characterized in that: In step 8, when it is necessary to combine different levels of knowledge expression to make more accurate and effective recommendations to users, a multimodal fusion algorithm is used to fuse the features of different modalities, and the intelligent agent is trained based on the fused features to generate a recommendation list L3 with multimodal expression for the same object.

Images