Multimodal-based recommendation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a recommendation method and device based on multiple modes, computer equipment, a readable storage medium and a program product. Constructing user interaction information and a product positive example diagram and a product negative example diagram in each modal based on the multi-modal data set, performing comparative learning according to the product positive example diagram and the product negative example diagram in each modal, obtaining product modal features in each modal, and performing fusion processing on the product modal features in each modal to obtain a product modal feature fusion model; and carrying out BPR negative example sampling based on the product negative example graph, and carrying out training and recommendation by adopting a BPR loss function according to the user interaction characteristics corresponding to the user interaction information and the multi-modal fusion characteristics. According to the method, a multi-modal information positive and negative example graph is constructed, more and more reliable multi-modal information is acquired by adopting comparative learning, and reliable negative examples are acquired based on the multi-modal information for sampling of BPR loss, so that the stability of BPR training is ensured, and the recommendation reliability is improved based on the idea of the reliable negative examples.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a multimodal recommendation method, apparatus, computer device, readable storage medium and program product. Background Art

[0002] Big data refers to massive and complex data sets that cannot be processed by traditional database software. It usually has the characteristics of large data volume, diverse data types, fast data generation speed, and low data value density. With the development of Internet and Internet of Things technology, the application scope of big data has been continuously expanded, covering many fields such as finance, medical care, transportation, and retail. Big data technology includes data collection, storage, processing, and analysis, and uses distributed computing and parallel processing to extract valuable information from massive data. Commonly used big data processing tools and platforms include Hadoop and Spark, which can efficiently process and analyze large-scale data sets. Through big data analysis, enterprises and organizations can achieve precision marketing, risk management, and operational optimization, and improve competitiveness and decision-making efficiency.

[0003] The recommendation system is one of the important applications based on big data technology and is widely used in e-commerce, social media and other fields. Its main function is to automatically recommend content or products that match users' interests based on their historical behavior data, preference information and product features. However, traditional recommendation systems have the problem of low reliability when users have few historical interaction records. Summary of the invention

[0004] Based on this, it is necessary to provide a multimodal recommendation method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the reliability of recommendation in response to the above technical problems.

[0005] In a first aspect, the present application provides a multimodal recommendation method, comprising:

[0006] Constructing user interaction information and a positive product graph and a negative product graph under each modality based on a multimodal dataset, wherein the positive product graph is determined based on the similarity of the product in the multimodal dataset under the corresponding modality, and the negative product graph is determined based on the negative similarity of the product in the multimodal dataset under the corresponding modality;

[0007] Compare and learn the positive and negative product images under each mode to obtain the product modal features under each mode.

[0008] The product modal features under each mode are fused to obtain multi-modal fusion features;

[0009] BPR negative example sampling is performed based on the product negative example graph, and training and recommendation are performed using a BPR loss function according to the user interaction features corresponding to the user interaction information and the multimodal fusion features.

[0010] In one of the embodiments, the modality includes a text modality and a picture modality; before the comparative learning based on the product positive example images and the product negative example images in each modality, the method further includes: denoising the product positive example images in the image modality based on the product negative example images in the text modality to obtain the denoised product positive example images in the image modality; denoising the product positive example images in the text modality based on the product negative example images in the picture modality to obtain the denoised product positive example images in the text modality.

[0011] In one of the embodiments, the comparative learning based on the product positive example image and the product negative example image in each modality includes: obtaining first feature information of the product positive example image after denoising in the same modality, and second feature information of the product negative example image; performing comparative learning based on the first feature information and the second feature information to obtain the relationship between the product positive example image and the product negative example image.

[0012] In one of the embodiments, the obtaining of product modal features under each modality includes: based on product positive example images under different modalities, using the LightGCN algorithm to obtain product embeddings of positive example co-occurrence relationships under different modalities; based on product negative example images under different modalities, using the LightGCN algorithm to obtain product embeddings of negative example co-occurrence relationships under different modalities.

[0013] In one of the embodiments, the product modal features under each modality are fused to obtain multimodal fusion features, including: determining the proportion of redundant information in each modality according to the product modal features under each modality; determining the total amount of redundant information of each modality based on the product modal features under each modality and the corresponding proportion of redundant information; determining the feature information of each modality based on the product modal features under each modality and the total amount of redundant information; determining the multimodal fusion features according to the feature information of each modality and the total amount of redundant information.

[0014] In one of the embodiments, constructing user interaction information and product positive and negative examples in each modality based on a multimodal dataset includes: preprocessing the multimodal dataset to obtain images and texts of all products in the multimodal dataset, as well as user interaction information; determining user characteristics based on the user interaction information; and determining product positive and negative examples in image modality from images of all products according to the user characteristics, and determining product positive and negative examples in text modality from texts of all products.

[0015] In a second aspect, the present application further provides a multimodal recommendation device, the device comprising:

[0016] A data processing module, used to construct user interaction information and a positive product example image and a negative product example image under each modality based on a multimodal data set, wherein the positive product example image is determined based on the similarity of the product in the multimodal data set under the corresponding modality, and the negative product example image is determined based on the negative similarity of the product in the multimodal data set under the corresponding modality;

[0017] The feature acquisition module is used to perform comparative learning based on the positive and negative product images under each modality to obtain the product modal features under each modality;

[0018] The feature fusion module is used to fuse the product modal features under each modality to obtain multi-modal fusion features;

[0019] The recommendation module is used to perform BPR negative example sampling based on the product negative example graph, and to perform training and recommendation using a BPR loss function according to the user interaction features corresponding to the user interaction information and the multimodal fusion features.

[0020] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0022] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0023] The above-mentioned recommendation method, device, computer equipment, computer-readable storage medium and computer program product based on multimodal data sets construct user interaction information and product positive and negative examples in each modality, perform comparative learning based on the product positive and negative examples in each modality, obtain product modal features in each modality, and fuse the product modal features in each modality to obtain multimodal fusion features, perform BPR negative example sampling based on product negative example images, and use BPR loss function for training and recommendation based on user interaction features and multimodal fusion features corresponding to user interaction information. It constructs multimodal information positive and negative example images, and uses comparative learning to clarify the relationship between the product and its positive and negative examples to obtain more and more reliable multimodal information related to the product, and obtains reliable negative examples from multimodal information for BPR loss sampling, so as to limit negative example sampling to a group of products that users do not like to ensure the stability of BPR training, thereby realizing the idea of ​​reliable negative examples to improve multimodal recommendations based on images and texts and improve the reliability of recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of the similarity of products in one embodiment;

[0026] Figure 2 is a flowchart of a multi-modal recommendation method in one embodiment;

[0027] Figure 3 is a schematic diagram of the principle of a recommendation method based on multimodality in one embodiment;

[0028] Figure 4 is a structural block diagram of a multi-modal recommendation device in one embodiment;

[0029] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] The core technologies of recommendation systems generally include collaborative filtering, content filtering, and hybrid recommendation methods. The effectiveness of recommendation systems depends on in-depth analysis of user behavior data and product information, which requires the system to have strong data processing and analysis capabilities. With the support of big data technology, recommendation systems can usually process massive amounts of data and quickly respond to user needs to improve user experience and satisfaction.

[0032] In traditional technology, the recommendation system mainly consists of the following parts:

[0033] Data collection: Collect users’ historical behavior data (such as product browsing history, product purchase history, ratings, etc.) and product feature information (such as title, description, category, etc.).

[0034] Data preprocessing: Clean, deduplicate, and normalize the collected data to ensure data quality and consistency.

[0035] Recommendation algorithm: Use collaborative filtering, content filtering or hybrid recommendation algorithms to calculate the similarity or correlation between users and products and generate a recommendation list.

[0036] Recommendation generation: Based on the calculation results of the recommendation algorithm, a recommendation list is generated and displayed to the user.

[0037] Feedback and evaluation: Collect user feedback on recommendation results, evaluate the effectiveness of the recommendation system, and continuously optimize and improve the recommendation algorithm.

[0038] However, traditional recommendation systems suffer from some obvious limitations:

[0039] Data sparsity: There are no direct interaction records between many users and products, which makes it difficult for recommendation algorithms to find enough similar users or items.

[0040] Cold start problem: For new users or new items, due to the lack of historical data, traditional recommendation systems find it difficult to give accurate recommendation results.

[0041] Single modality limitation: Traditional recommendation systems usually only utilize a single type of data (such as user behavior data or product feature data) and fail to make full use of other valuable information.

[0042] Insufficient recommendation diversity: Traditional recommendation systems may over-focus on users' existing areas of interest, resulting in a lack of diversity in recommendation results and a decline in user experience.

[0043] In order to overcome the shortcomings of traditional recommendation systems, multimodal recommendation systems have emerged. Multimodal recommendation systems use multiple types of data sources (modalities), such as text, images, audio, video, etc., and comprehensively consider information from different modalities to provide users with more comprehensive and personalized recommendations. Multimodal recommendation systems combine multiple types of data, making full use of information such as text descriptions, image content, and user comments to improve the accuracy of recommendation results. By introducing multimodal data, the system can use information from other modalities for recommendations even when there are few records of user and item interactions. For example, new users and new products can be initially recommended through multimodal data (such as users' social media information or the form, purpose, and description of products), thereby alleviating the impact of the cold start problem. Multimodal recommendation systems can comprehensively consider the various needs of users, provide richer and more diverse recommendation results, and improve user experience. The fusion of multimodal data enables the recommendation system to maintain a high recommendation performance when facing noise or missing single modal data, thereby enhancing the robustness of the system. Through multimodal recommendation systems, users can obtain more personalized, accurate, and diverse recommendation services, improve the user experience, and increase user stickiness.

[0044] However, multimodal recommendation also faces unique challenges. It is expected to extract effective information from multimodal information and effectively integrate it with the ID-based information mined by traditional recommendation algorithms. However, in reality, multimodal information contains a lot of redundant and interfering noise information, such as Figure 1 As shown in the figure, the backgrounds of the two product images are too similar, resulting in a high degree of similarity in the display of two completely different clothing products. Similarly, in the product descriptions, "woman" and "girl" have similar semantics, which also leads to a high degree of similarity in the text descriptions of the two products (e.g., 64.8%). Therefore, multimodal information is not necessarily reliable.

[0045] Based on this, the embodiment of the present application provides a multimodal recommendation method, which improves multimodal recommendations based on images and texts through the idea of ​​reliable negative examples. It constructs user interaction information and product positive and negative examples in each modality based on a multimodal data set, and performs comparative learning based on the product positive and negative examples in each modality, and makes it close to the positive example by increasing the difference between the product and its negative example, and obtains the product modal features in each modality, and fuses the product modal features in each modality to obtain multimodal fusion features, and then performs BPR (Bayesian Personalized Ranking, a sorting algorithm for recommendation systems) negative example sampling based on the product negative example map, and uses the BPR loss function for training and recommendation based on the user interaction features and multimodal fusion features corresponding to the user interaction information. Since it performs BPR negative example sampling based on the product negative example map, the stability of BPR training is ensured by limiting the negative example sampling to the product negative example map that the user does not like, thereby improving the reliability of the recommendation.

[0046] In an exemplary embodiment, Figure 2 As shown, a multimodal recommendation method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method can include the following steps:

[0047] Step 202: construct user interaction information and product positive and negative examples in each modality based on the multimodal dataset.

[0048] Among them, the multimodal dataset can be a data source including multiple types (ie, modalities, such as text, pictures, etc.) of data. For example, in a product recommendation scenario, the multimodal dataset can be text information and picture information of the product being sold. In this embodiment, taking the multimodal dataset as product review data as an example, since the review data is generally a user's evaluation of the goods that have been sold, it usually includes picture information of the goods that have been sold, introduction information of the goods, and user information of purchasing and commenting on the goods. Therefore, based on the multimodal dataset, user interaction information and product positive and negative examples in each modality can be constructed.

[0049] Specifically, the user interaction information can be determined based on the interaction relationship of users in the multimodal dataset. The product positive example image can be determined based on the similarity of the product in the multimodal dataset in the corresponding modality, and the product negative example image can be determined based on the negative similarity of the product in the multimodal dataset in the corresponding modality.

[0050] In one embodiment, user interaction information and positive and negative product images in each modality are constructed based on a multimodal dataset, which may specifically include: preprocessing the multimodal dataset, obtaining image information and text information of all products in the multimodal dataset, and obtaining user interaction information in the multimodal dataset; determining user embedding based on the user interaction information; determining positive and negative product images in the image modality from the images of all products according to the user embedding, and determining positive and negative product images in the text modality from the texts of all products.

[0051] In this embodiment, still taking the product review data as an example of a multimodal data set, the product review data is first pre-processed, such as removing products with less than 5 interactions and users with less than 5 interactions, to obtain a basic data set. This embodiment takes the modality including text modality and image modality as an example, based on which the image information and text information of all products can be extracted respectively to construct data sets of different modalities, that is, product image data sets and product text data sets are constructed respectively. Then, based on CLIP (Contrastive Language-Image Pre-training, a large model-based embedding method), the semantics of the product image and the semantics of the product text are obtained, and then the product image and product text can be embedded in the same space as the user ID.

[0052] The establishment of user interaction information relies on the interaction relationship of users. For example, an edge can be established between each user and the product they have interacted with, thereby forming a heterogeneous graph based on user interaction, that is, constructing user interaction information. Then, based on the user interaction information, user characteristics can be determined. Specifically, the Light GCN (Light Graph Convolutional Networks) algorithm can be used to directly perform graph convolution on the user interaction information, that is, the heterogeneous graph of user interaction, so as to learn the user embedding and product embedding from the co-occurrence relationship of user interaction to determine user characteristics.

[0053] In one scenario, the weights of the edges in the heterogeneous graph of user interactions can also be regularized according to the Light GCN method, thereby ensuring that after each graph convolution, the user embedding and product embedding are controlled within a certain size.

[0054] Furthermore, the positive and negative product images of the image modality can be determined from the image dataset of the product constructed above according to the user characteristics, and the positive and negative product images of the text modality can be determined from the text dataset of the product constructed above.

[0055] Among them, the positive example graph can be composed of edges established based on the K most similar products in the corresponding modality determined by the K-nearest neighbor algorithm. The negative example graph can be composed of edges established based on the K most dissimilar products in the corresponding modality determined by the K-nearest neighbor algorithm. Specifically, based on the semantics of the product images and the semantics of the product texts obtained above, the image similarity and text similarity of all products and user interaction products can be calculated. Specifically, the K-nearest neighbor algorithm can be used to establish edges only for the product and its K most similar products to obtain a multimodal positive example graph (i.e., a positive example graph of the image modality and a positive example graph of the text modality). The weight of the edge is initialized to the similarity, and the same regularization strategy can also be used to adjust the weight. The construction method of the negative example graph is similar to that of the positive example graph, except that when the K-nearest neighbor algorithm is used, it is based on negative similarity, that is, the K edges retained for the product are established by all products and their K most dissimilar products.

[0056] When the similarity between two products reaches 99%, it is almost certain that the two products are not affected by noise and achieve "true" similarity. And after performing similarity distribution analysis during the experiment, it was found that there are very few products with a similarity of 99%, so the information provided by "true" similar products is very little and insufficient to be used. However, the experiment found that there are many products with a similarity of about 1%. Based on this, this embodiment considers from the perspective of negative example learning that a large amount of negative example information can also bring good results for the learning of product features. For example, if you don't know which other product a certain product is similar to, but you know that a large number of optional products are not similar, therefore, in the process of product feature learning, the relevant information of these dissimilar products can be made farther apart in the feature space, thereby moving the product to its most ideal embedding position in the feature space to a certain extent. Therefore, this embodiment improves the multimodal recommendation of images and texts by constructing a reliable negative example graph.

[0057] Step 204 , performing comparative learning based on the positive product image and the negative product image under each modality, and obtaining the modal features of the product under each modality.

[0058] Specifically, the first feature information of the positive example image of the product under the same mode and the second feature information of the negative example image of the product are obtained; and comparative learning is performed based on the first feature information and the second feature information to obtain the relationship between the positive example image of the product and the negative example image of the product.

[0059] Since the multimodal positive graph stores the K products that are most similar to the product itself, and the multimodal negative graph stores the K products that are most dissimilar. By using the comparison between negative and positive examples, the model can more easily distinguish the similarities and associations between different products, and learn more robust embeddings for products and users. Specifically, the information learned from the positive graph, i.e., the first feature information, can be regarded as positive information, the information in the negative graph, i.e., the second feature information, can be regarded as negative information, and the information learned from the two graphs, i.e., the first feature information and the second feature information, can be regarded as negative example pairs. By applying contrastive learning methods, the difference between the product and its negative example is increased to make it close to the positive example, thereby clarifying the relationship between the product and its positive and negative examples, and obtaining more and more reliable multimodal information related to the product, so that the learned product embedding is more reliable, that is, reliable knowledge can be learned from a large number of reliable dissimilar items.

[0060] In one scenario, in order to enable the multimodal positive example images of images and texts to retain a more reliable connection and avoid learning information from noise edges, before comparative learning is performed based on the product positive example images and product negative example images in each modality, the method may also include: denoising the product positive example images in the image modality based on the product negative example images in the text modality to obtain the denoised product positive example images in the image modality; denoising the product positive example images in the text modality based on the product negative example images in the image modality to obtain the denoised product positive example images in the text modality.

[0061] Since the similarity graph constructed above, i.e., the positive example graph, may contain some noise edges, such as edges with a similarity of 20%-30%, when the similarity reaches this value, there is every reason to believe that these edges are noise edges. In this embodiment, in order to prune the noise edges, the negative example graph constructed by the two modes of picture and text can be used to supervise each other, and the multimodal positive example graph can be pruned according to the multimodal negative example graph, thereby retaining the most credible multimodal connection. In the example, when the negative example graph of the picture modality and the positive example graph of the text modality have overlapping edges, the text positive example graph can be pruned, that is, these edges can be removed from the text positive example graph. Similarly, when the negative example graph of the text modality and the positive example graph of the picture modality have overlapping edges, the picture positive example graph can be pruned, that is, these edges can be removed from the picture positive example graph, thereby obtaining a multimodal information positive example graph with less noise information.

[0062] Furthermore, when performing comparative learning based on the positive product images and negative product images in each modality, comparative learning can be performed based on the positive product images in the denoised image modality and the positive product images in the denoised text modality and the negative product images in the corresponding modality to further improve the accuracy of recommendations.

[0063] The product modal feature can be the product embedding under the corresponding modality. Specifically, based on the product positive example images under different modalities, the LightGCN algorithm can be used to obtain the product embedding of the positive example co-occurrence relationship under different modalities; based on the product negative example images under different modalities, the LightGCN algorithm can be used to obtain the product embedding of the negative example co-occurrence relationship under different modalities, so as to obtain the product modal features under each modality. Specifically, the LightGCN algorithm can be used to perform graph convolution on the product positive example images under different modalities respectively, so as to obtain two product embeddings from the positive example co-occurrence relationship of multi-modal information. By using the LightGCN algorithm to perform graph convolution on the product negative example images under different modalities respectively, two product embeddings from the negative example co-occurrence relationship of multi-modal information are obtained.

[0064] Step 206 , fusing the product modal features under each modality to obtain a multi-modal fusion feature.

[0065] The multimodal fusion feature refers to the final product feature obtained by fusing the features of the product under different modes. Specifically, the multimodal fusion feature is obtained by fusing the product modal features under each mode.

[0066] Step 208: perform BPR negative example sampling based on the product negative example graph, and perform training and recommendation.

[0067] Specifically, BPR negative example sampling is performed based on the product negative example graph, and training is performed using the BPR loss function according to the user interaction features and multimodal fusion features corresponding to the user interaction information, and then product recommendations can be made. That is, by inputting user information into the obtained tensor, the corresponding result tensor is obtained, which can be converted into the user's preferred products for recommendation.

[0068] Since the training concept of the traditional BPR loss function (Bayesian Personalized Ranking loss, referred to as BPRloss) is to expand the difference between the similarity between the user's embedding and the embedding of the user's interaction and the similarity between the products that the user has not interacted with. This means that products that the user has not interacted with are uniformly considered as negative examples, but in real life, it does not mean that users are not interested in products that they have not purchased. Therefore, the traditional BPR loss is not stable, or its stability is determined by the reliability of the selection of negative examples.

[0069] In this embodiment, the introduction of multimodal information provides an opportunity to find reliable negative examples. Specifically, the user's embedding can be initialized from the multimodal information of the products with which the user has interacted. However, the similarity between the user embedding and the product's image and text is calculated, and the products with a similarity of less than 10% are taken as stable negative examples of the user, and the negative example sampling of BPR loss is limited to these products, making the learning process of BPR loss more stable and reliable. It obtains reliable negative examples from multimodal information for BPR loss sampling, and limits the negative example sampling to a group of products that users do not like to ensure the stability of BPR loss training. The method of pruning multimodal graphs using negative examples and the method of assisting BPR loss training have good transferability, and any multimodal recommendation model can adopt the same method to improve performance.

[0070] In the above-mentioned multimodal recommendation method, user interaction information and product positive and negative examples in each modality are constructed based on the multimodal data set, and comparative learning is performed based on the product positive and negative examples in each modality to obtain the product modal features in each modality, and the product modal features in each modality are fused to obtain multimodal fusion features, and BPR negative example sampling is performed based on the product negative example map, and training and recommendation are performed using the BPR loss function based on the user interaction features and multimodal fusion features corresponding to the user interaction information. It constructs multimodal information positive and negative example maps, and uses comparative learning to clarify the relationship between the product and its positive and negative examples, so as to obtain more and more reliable multimodal information related to the product, and obtains reliable negative examples from the multimodal information for BPR loss sampling, so as to limit the negative example sampling to a group of products that users do not like to ensure the stability of BPR training, thereby realizing the idea of ​​reliable negative examples to improve the multimodal recommendation based on images and texts and improve the reliability of recommendations.

[0071] In an exemplary embodiment, the implementation principle of the multimodal recommendation method is as follows: Figure 3 As shown, it may specifically include:

[0072] First, by preprocessing the multimodal dataset, the image information, text information and user interaction information of all products in the multimodal dataset are obtained. Among them, the multimodal dataset can be product sales data, comment data, etc., which can specifically include product display pictures, product introductions, descriptions, and interactive relationships with users. In order to embed product pictures and product texts into the same space as the user ID, CLIP can be used to obtain the image semantics of each product picture to construct the product picture dataset, and the text semantics of the first product text can be obtained to construct the product text dataset, thereby obtaining a graph based on two modal information, that is, a product picture graph composed of a picture dataset and a product text graph composed of a text dataset.

[0073] The establishment of user interaction information relies on the interaction relationship of users. For example, an edge can be established between each user and the product they have interacted with, thereby forming a heterogeneous graph based on user interaction, that is, constructing user interaction information (also called user interaction graph). In addition, user characteristics can be determined based on user interaction information. Specifically, the Light GCN algorithm can be used to directly perform graph convolution on user interaction information, that is, the heterogeneous graph of user interaction, so as to learn user embedding and product embedding from the co-occurrence relationship of user interaction to determine user characteristics.

[0074] Furthermore, according to the user characteristics, the positive and negative image examples of the image modality can be determined from the image dataset of the product constructed above, and the positive and negative text examples of the text modality can be determined from the text dataset of the product constructed above.

[0075] Specifically, the positive example graph can be composed of the edges of the K most similar products in the corresponding modality determined by the K-nearest neighbor algorithm. The negative example graph can be composed of the edges of the K most dissimilar products in the corresponding modality determined by the K-nearest neighbor algorithm. Specifically, based on the image semantics of the product images and the text semantics of the product texts obtained above, the image similarity of all product images and user interactive products, as well as the similarity of all product texts and user interactive products and texts can be calculated. Specifically, the K-nearest neighbor algorithm can be used to only establish edges for the product and its K most similar products to obtain a multimodal positive example graph (i.e., a positive example graph of the image modality and a positive example graph of the text modality). The weights of the edges are initialized to the similarity, and the same regularization strategy can be used to adjust the weights. The construction method of the negative example graph is similar to that of the positive example graph, except that when the K-nearest neighbor algorithm is used, it is based on the negative similarity, that is, the K edges retained for the product are established by all products and their K most dissimilar products. Thus, the positive and negative example images in the image mode are obtained, and the positive and negative example images in the text mode are obtained.

[0076] In order to enable the multimodal positive example images of images and texts to retain a more reliable connection, the positive example images of the products in the image modality can also be denoised based on the negative example images of the products in the text modality to obtain the denoised positive example images of the products in the image modality. That is, when the negative example images of the text modality and the positive example images of the image modality have overlapping edges, the positive example images of the image modality can be pruned, that is, these edges can be removed from the positive example images of the image modality, thereby obtaining the positive example images of the image modality with less noise information. Similarly, the positive example images of the products in the text modality can be denoised based on the negative example images of the products in the image modality to obtain the denoised positive example images of the products in the text modality. That is, when the negative example images of the image modality and the positive example images of the text modality have overlapping edges, the positive example images of the text modality can be pruned, that is, these edges can be removed from the positive example images of the text modality, thereby obtaining the positive example images of the text modality with less noise information.

[0077] Then, the product positive and negative images after denoising in each modality are compared and learned. Since the identification of an object can be assisted by information learned from objects similar to the object, or by objects completely dissimilar to the object. Based on this, the information learned from the multimodal positive image can be identified as information completely similar to the product, and the information learned from the multimodal negative image can be identified as information completely dissimilar to the product, and the two constitute negative learning of comparative learning. For example, for the image modality, the first feature information of the product positive image after denoising in the modality and the second feature information of the product negative image are obtained, and comparative learning is performed based on the first feature information and the second feature information, so as to increase the difference between the product and its negative example to make it close to the positive example, so as to clarify the relationship between the product and its positive and negative examples, so that the model can more easily distinguish the similarity and correlation between different products. Similarly, for the text modality, the first feature information of the product positive image after denoising in the modality and the second feature information of the product negative image are obtained, and comparative learning is performed based on the first feature information and the second feature information, so as to achieve reliable knowledge from a large number of reliable dissimilar objects.

[0078] Since the user interaction graph is a heterogeneous graph, and the multimodal positive and negative example graphs (four graphs, including positive and negative example graphs of pictures and texts) are all product graphs with only products, we can use the LightGCN algorithm to directly perform graph convolution on the user interaction graph, and learn user embedding and product embedding from the co-occurrence relationship of user interactions.

[0079]

[0080] This formula represents the average pooling of the results of each layer of graph convolution, that is, Represents the final Embedding obtained after average pooling, is the embedding of the lth convolution layer in the user interaction graph. L represents the total number of convolutions. In this embodiment, L can be set to 2. Its advantage is that it is equivalent to establishing a residual connection to prevent the problem of over-smoothing caused by too fast graph convolution.

[0081] The LightGCN algorithm can also be used to perform graph convolution on multi-modal positive and negative example images to obtain product modal features under each mode:

[0082]

[0083] in, A graph representing the modality positive and negative examples corresponding to the modality m (such as text modality, image modality), It represents the embedding of product i before convolution on the graph. It represents the embedding of product i after convolution on the graph (since this embodiment is set to convolve only once on the multimodal information graph, the number of convolutions l is not set) and the convolution of the four multimodal graphs, that is, the modal features of the products under each mode. This formula represents a single graph convolution operation, that is, directly performing matrix multiplication with the Embedding matrix and the regularized adjacency matrix, which represents one convolution. From the positive example graph, two product embeddings from the co-occurrence relationship of the positive example of multimodal information and two product embeddings from the co-occurrence relationship of the negative example of multimodal information can be learned.

[0084] Then, the product modal features under each mode are fused to obtain multimodal fusion features:

[0085]

[0086]

[0087] This formula represents the multimodal information fusion operation for removing redundancy. Multimodal information fusion is easy to understand. For example, if you want to represent the characteristics of a product, you must generate a comprehensive description for it based on the graphic information of the product. Therefore, multimodal information fusion is needed. Redundancy removal is because multimodal information often has some information overlap. For example, in the text description, it may appear: This is a pair of blue washed jeans..., and the picture shows the jeans, and it can be seen that they are also blue. If the information of the two is combined to make a description, it will definitely not say twice that this is a blue jeans, but will naturally calculate the redundant information of blue only once. The above formula simulates this process. The first three steps (the first 3 formulas) represent the operation of filtering redundant information.

[0088] Specifically, the first step is to find out the proportion of redundant information in each mode, that is, to determine the proportion of redundant information in each mode according to the product modal characteristics under each mode. mrepresents the proportion of redundant information in mode m, W4 and b4 are conventional parameters of linear transformation, and q1 represents linear mapping. Therefore, Formula 1 represents (i.e., the sum of the embeddings corresponding to users and products in modality m) is mapped to the redundant information space, and based on the values ​​in the space, the proportion of redundant information is calculated.

[0089] The second step is to integrate the redundant information in each mode, that is, based on the product modal characteristics under each mode and the proportion of corresponding redundant information, determine the total amount of redundant information in each mode. Es represents the sum of all redundant information obtained by integrating the redundant information in multiple modes.

[0090] The third step represents the information with its own characteristics obtained after subtracting redundant information from each mode. It is based on the product modal characteristics under each mode and the total amount of redundant information to determine the characteristic information of each mode. It represents the characteristic information obtained after subtracting redundant information under mode m.

[0091] The fourth step represents the determination of multimodal fusion features based on the feature information of each modality and the total amount of redundant information. mul It represents the final embedding based on multimodal information calculated by integrating the characteristic information and redundant information between each modality. M represents all modalities. In this embodiment, M = {text, image}, because only the image and text modality is included. Pm represents the weight of the importance of multimodal information. It is a projection operation, which is used to adjust the proportion of each modal information in the final Embedding. This is because image and text information are not equally important. In actual tasks, text information is usually more important.

[0092] This is a common information fusion method that converts the combination of two parts of information into a learnable function and mines the common part from the two parts of information to prevent over-learning of the shared information between the two. The final result is the embedding E of the multimodal information of all product images and texts. mul Then we need to combine it with the user embedding E learned from the user interaction graph id To carry out fusion, the specific method can be to add and sum, then:

[0093]

[0094] Depend on Figure 3 It can be seen that user interaction information can construct a neural network that includes user nodes and product nodes connected to each other, so we can get an embedding of the interactive product learned based on the user interaction graph. and the user's embed i,mul .ei It is a combination of the embedding of interactive products learned based on the user interaction graph and the user's embed i,mul , which represents the final user embedding.

[0095] For the product multimodal information above, since it only contains multimodal information between products, in order to construct the same tensor, a user-based embedding is also required. Therefore, the user embedding learned from the user interaction information is average pooled to obtain the result e u,mul , The product modal features under each mode are fused, and the multimodal fusion feature of the product is obtained, namely E mul .e u Represents the final product embedding obtained by integrating two information sources: the user interaction graph and the product's multimodal information graph.

[0096] where e u,mul The specific calculation formula is:

[0097]

[0098] This formula represents the initialization of user embedding, which is the average pooling of multimodal information embedding of products that users have interacted with. Since multimodal information can only construct a K-nearest neighbor graph between products, that is, focusing on I, there is no user node in this graph, so the user embedding needs to be calculated. This calculation method takes into account the regularization between user nodes and product nodes. Nu represents how many users have interacted with the product. If Nu is large, it means that the product is a popular product. In order to increase user differentiation, the influence of popular products can be reduced, so it is divided by Nu. Ni is the product that users have interacted with, representing the average pooling of each product. represents the embedding of product i after convolution on the graph, as mentioned above

[0099] By calculating the similarity between user embedding and product images, products with similarity less than 10% are taken as stable negative examples for users, and the negative example sampling of BPR loss is limited to these products for BPR training, making the learning process of BPR loss more stable and reliable. It obtains reliable negative examples from multimodal information for BPR loss sampling, and limits negative example sampling to a group of products that users do not like to ensure the stability of BPR loss training. The method of pruning multimodal graphs using negative examples and the method of assisting BPR loss training have good transferability, so that any multimodal recommendation model can adopt the same method to improve performance.

[0100] Finally, we can directly recall the 20 products with the highest similarity based on the calculated user embedding and product embedding, and judge the recall precision, i.e. the accuracy of the recommendation, based on the loss function Recall@10 (used to measure the ability of the model to correctly identify relevant results in the first 10 prediction results, representing the proportion of correct answers in the first 10 results), Recall@20 (used to measure the ability of the model to correctly identify relevant results in the first 20 prediction results, representing the proportion of correct answers in the first 20 results), NDCG@10 (used to measure the performance of the system in the first 10 results, i.e., the quality of the first 10 results returned by the system), and NDCG@20 (used to measure the performance of the system in the first 20 results, i.e., the quality of the first 20 results returned by the system). Specifically, the recommendation effect can be evaluated according to the pre-set evaluation criteria. These evaluation criteria can include multiple indicators such as click-through rate, conversion rate, user satisfaction, etc. to comprehensively evaluate the performance and effect of the recommendation system. Experiments have shown that, in terms of various evaluation indicators, its recommendation accuracy has achieved a significant improvement at the percentage level compared to the multimodal recommendation methods commonly used in traditional technologies.

[0101] In practical applications, the neural network can be obtained by training these data in the above manner according to the image information, text information and user interaction information of relevant products (such as products that have been sold or products to be recommended). Specifically, data cleaning, feature extraction and data format conversion can be performed to ensure that the data is suitable for the input requirements of the model. At the same time, the input data of each batch can also be set to ensure that the model can efficiently process large-scale data. During the operation of the model, user data can be input into the model in batches. The model generates user embedding and product embedding through calculation. These embedding vectors contain multimodal feature information of users and products, which are the basis for the recommendation system to make recommendations. Based on the generated user embedding and product embedding, the model can make personalized recommendations to users, and then the recommendation effect can be evaluated according to the pre-set evaluation criteria. When the recommendation effect meets the target requirements, it means that the network can be applied to actual recommendations. In actual recommendations, by inputting user information into the obtained network, the corresponding features are obtained, which can be converted into the user's preferred products for recommendation. After that, more user interaction information and product multimodal information can be continuously added for learning, and the recommendation results for existing users can be continuously corrected through learning of large amounts of data, making the model more accurate.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a multimodal recommendation device for implementing the multimodal recommendation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more multimodal recommendation device embodiments provided below can refer to the limitations of the multimodal recommendation method above, and will not be repeated here.

[0104] In an exemplary embodiment, Figure 4 As shown, a multi-modal recommendation device is provided, comprising: a data processing module 402, a feature acquisition module 404, a feature fusion module 406 and a recommendation module 408, wherein:

[0105] A data processing module 402 is used to construct user interaction information and a positive example image and a negative example image of a product in each modality based on the multimodal dataset, wherein the positive example image of the product is determined based on the similarity of the product in the multimodal dataset in the corresponding modality, and the negative example image of the product is determined based on the negative similarity of the product in the multimodal dataset in the corresponding modality;

[0106] A feature acquisition module 404 is used to perform comparative learning based on the positive product image and the negative product image under each mode to obtain the product modal features under each mode;

[0107] The feature fusion module 406 is used to fuse the product modal features under each modality to obtain a multi-modal fusion feature;

[0108] The recommendation module 408 is used to perform BPR negative example sampling based on the product negative example graph, and to perform training and recommendation using a BPR loss function according to the user interaction features corresponding to the user interaction information and the multimodal fusion features.

[0109] In an exemplary embodiment, the modality includes a text modality and a picture modality; the device also includes a denoising processing module, which is used to denoise the product positive example image of the image modality based on the product negative example image of the text modality to obtain the denoised product positive example image of the image modality; and denoise the product positive example image of the text modality based on the product negative example image of the picture modality to obtain the denoised product positive example image of the text modality.

[0110] In an exemplary embodiment, the device also includes a denoising processing module, which is used to: obtain the first feature information of the product positive example image after denoising in the same modality, and the second feature information of the product negative example image; perform comparative learning based on the first feature information and the second feature information to obtain the relationship between the product positive example image and the product negative example image.

[0111] In an exemplary embodiment, the feature acquisition module is also used to: based on the product positive example images under different modalities, use the LightGCN algorithm to obtain the product embedding of the positive example co-occurrence relationship under different modalities; based on the product negative example images under different modalities, use the LightGCN algorithm to obtain the product embedding of the negative example co-occurrence relationship under different modalities.

[0112] In an exemplary embodiment, the feature fusion module is also used to: determine the proportion of redundant information in each modality based on the product modal features under each modality; determine the total amount of redundant information of each modality based on the product modal features under each modality and the corresponding proportion of redundant information; determine the feature information of each modality based on the product modal features under each modality and the total amount of redundant information; determine the multimodal fusion features based on the feature information of each modality and the total amount of redundant information.

[0113] In an exemplary embodiment, the data processing module is also used to: preprocess the multimodal dataset to obtain images and texts of all products in the multimodal dataset, as well as user interaction information; determine user characteristics based on the user interaction information; and determine product positive and product negative examples in image modality from the images of all products according to the user characteristics, and determine product positive and product negative examples in text modality from the texts of all products.

[0114] Each module in the above-mentioned multimodal recommendation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0115] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a recommendation method based on multimodality is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0116] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0119] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal recommendation method, characterized in that: The method comprises: Constructing user interaction information and a positive product graph and a negative product graph under each modality based on a multimodal dataset, wherein the positive product graph is determined based on the similarity of the product in the multimodal dataset under the corresponding modality, and the negative product graph is determined based on the negative similarity of the product in the multimodal dataset under the corresponding modality; Compare and learn the positive and negative product images under each mode to obtain the product modal features under each mode. The product modal features under each mode are fused to obtain multi-modal fusion features; BPR negative example sampling is performed based on the product negative example graph, and training and recommendation are performed using a BPR loss function according to the user interaction features corresponding to the user interaction information and the multimodal fusion features.

2. The method according to claim 1, characterized in that The modalities include text modalities and image modalities; before performing comparative learning based on the positive product example images and the negative product example images under each modality, the method further includes: De-noising the positive product image in the image modality based on the negative product image in the text modality to obtain a denoised positive product image in the image modality; The product positive example image in the text modality is denoised based on the product negative example image in the picture modality to obtain the denoised product positive example image in the text modality.

3. The method according to claim 2, characterized in that The comparative learning based on the positive product example image and the negative product example image under each modality includes: Obtain the first feature information of the product positive example image after denoising in the same mode, and the second feature information of the product negative example image; Comparative learning is performed based on the first feature information and the second feature information to obtain a relationship between the positive example image of the product and the negative example image of the product.

4. The method according to claim 2, characterized in that: The obtaining of product modal features in each modality includes: Based on the product positive example graphs under different modalities, the LightGCN algorithm is used to obtain the product embedding of the positive example co-occurrence relationship under different modalities; Based on the negative example graphs of products under different modalities, the LightGCN algorithm is used to obtain the product embedding of the negative example co-occurrence relationship under different modalities.

5. The method according to claim 1, characterized in that The product modal features under each modality are fused to obtain multi-modal fusion features, including: According to the product modal characteristics under each mode, determine the proportion of redundant information in each mode; Based on the product modal characteristics under each mode and the proportion of corresponding redundant information, determine the total amount of redundant information of each mode; Determine feature information of each mode based on the product modal features under each mode and the total amount of redundant information; The multimodal fusion feature is determined according to the feature information of each modality and the total amount of redundant information.

6. The method according to any one of claims 1 to 5, characterized in that: The method of constructing user interaction information and product positive and negative examples in each mode based on a multimodal dataset includes: Preprocessing the multimodal data set to obtain images and texts of all products in the multimodal data set, as well as user interaction information; Determining user characteristics based on the user interaction information; According to the user features, positive product examples and negative product examples in picture mode are determined from pictures of all products, and positive product examples and negative product examples in text mode are determined from texts of all products.

7. A multimodal recommendation device, characterized in that: The device comprises: A data processing module, used to construct user interaction information and a positive product example image and a negative product example image under each modality based on a multimodal data set, wherein the positive product example image is determined based on the similarity of the product in the multimodal data set under the corresponding modality, and the negative product example image is determined based on the negative similarity of the product in the multimodal data set under the corresponding modality; The feature acquisition module is used to perform comparative learning based on the positive and negative product images under each modality to obtain the product modal features under each modality; The feature fusion module is used to fuse the product modal features under each modality to obtain multi-modal fusion features; The recommendation module is used to perform BPR negative example sampling based on the product negative example graph, and to perform training and recommendation using a BPR loss function according to the user interaction features corresponding to the user interaction information and the multimodal fusion features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images