Multi-modal recommendation method and device, model training method and device, equipment and medium

Abstract

The invention discloses a multi-modal recommendation method and device, a model training method and device, equipment and a medium, and belongs to the field of multi-modal recommendation. The method comprises the following steps: acquiring multi-modal feature vectors of at least two to-be-recommended items; obtaining an object feature vector, wherein the object feature vector is used for indicating a preference feature vector of the to-be-recommended object; de-noising the multi-modal feature vectors of the at least two to-be-recommended items to obtain de-noised multi-modal feature vectors of the at least two to-be-recommended items; calculating preference scores of the to-be-recommended objects to each to-be-recommended item based on the de-noised multi-modal feature vectors and the object feature vectors; and generating a recommendation list of the to-be-recommended objects based on the preference score of each to-be-recommended item. According to the multi-modal recommendation method and device, the multi-modal feature vectors are firstly denoised, and the recommendation list is generated based on the denoised feature vectors, so that the accuracy of multi-modal recommendation can be improved. In addition, alignment is carried out in the multi-modal recommendation process, and the accuracy of multi-modal recommendation can also be improved.

Description

Technical Field

[0001] The present application relates to the field of multimodal recommendation, and particularly to a multimodal recommendation method, a model training method, an apparatus, a device, and a medium. Background Art

[0002] A multimodal recommendation system is a recommendation system that can process and utilize multiple modalities of data. With the rapid development of multimedia services, such as short videos, news, etc., the data to be processed in the recommendation system is becoming more and more diverse, including multiple modalities of data such as text, images, and audio. Therefore, multimodal recommendation systems have emerged and become a research hotspot in the current field of recommendation systems.

[0003] In the related art, when using multimodal data for recommendation, the processing effect of noise in multimodal content is not very good, which easily affects the accuracy of the final recommendation result.

[0004] Therefore, how to design a multimodal recommendation system to achieve a more accurate recommendation effect is a problem that needs to be solved currently. Summary of the Invention

[0005] The present application provides a multimodal recommendation method, a model training method, an apparatus, a device, and a medium. The technical solutions are as follows:

[0006] According to one aspect of the present application, there is provided a multimodal recommendation method, the method including:

[0007] Obtain multimodal feature vectors of at least two items to be recommended, where the multimodal feature vector of each item to be recommended includes feature vectors of each item to be recommended in at least two modalities; and obtain an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended;

[0008] Denoise the multimodal feature vectors of the at least two items to be recommended to obtain denoised multimodal feature vectors of the at least two items to be recommended;

[0009] Based on the denoised multimodal feature vectors and the object feature vector, calculate the preference scores of the object to be recommended for each of the items to be recommended;

[0010] Based on the preference scores of each of the items to be recommended, generate the recommendation list of the object to be recommended, where the recommendation list includes at least one of the items to be recommended.

[0011] According to one aspect of the present application, there is provided a multimodal recommendation method, where the denoised multimodal feature vector of the first item to be recommended includes denoised feature vectors of the first item to be recommended in at least two modalities;

[0012] Aggregating the multi-modal feature vectors of the first item to be recommended and the multi-modal feature vectors of the second item to be recommended based on the denoised similarity information to obtain the denoised multi-modal feature vector of the first item to be recommended, including:

[0013] Taking the similarity between the first item to be recommended and the second item to be recommended in the i-th modality as a weight, aggregating the feature vectors of the first item to be recommended and the second item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer;

[0014] Repeating the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0015] According to one aspect of the present application, a multi-modal recommendation method is provided. The denoised multi-modal feature vector of the first item to be recommended includes the denoised feature vectors of the first item to be recommended in at least two modalities;

[0016] Aggregating the multi-modal feature vectors of the first item to be recommended and the multi-modal feature vectors of the fourth item to be recommended based on the denoised correlation information to obtain the denoised multi-modal feature vector of the first item to be recommended, including:

[0017] Taking the similarity between the first item to be recommended and the fourth item to be recommended in the i-th modality as a weight, aggregating the feature vectors of the first item to be recommended and the fourth item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer;

[0018] Repeating the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0019] According to one aspect of the present application, a multi-modal recommendation method is provided. Obtaining the multi-modal feature vectors of at least two items to be recommended, and the multi-modal feature vector of each item to be recommended includes the feature vectors of each item to be recommended in at least two modalities, including:

[0020] Extracting the multi-modal feature vectors of the at least two items to be recommended based on a multi-modal model.

[0021] According to one aspect of the present application, a model training method for a multi-modal model is provided. The method includes:

[0022] Obtain the multi-modal feature vectors of at least two training samples, where the multi-modal feature vector of each training sample includes the feature vectors of the training sample in at least two modalities; and obtain an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object;

[0023] Denoise the multi-modal feature vectors of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples;

[0024] Based on the denoised multi-modal feature vectors and the object feature vector, calculate the predicted preference scores of the sample object for each of the training samples;

[0025] Train the multi-modal model based on the sample loss of the training samples.

[0026] According to one aspect of the present application, there is provided a method for training a multi-modal model. The denoising of the multi-modal feature vectors of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples includes:

[0027] Based on at least one of the similarity information and the correlation information of the at least two training samples, denoise the multi-modal feature vectors of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples;

[0028] Wherein, the similarity information is used to indicate the similarity degree of two training samples in the same modality, and the correlation information is used to indicate the correlation degree between two training samples.

[0029] According to one aspect of the present application, there is provided a method for training a multi-modal model. The denoising of the multi-modal feature vectors of the at least two training samples based on at least one of the similarity information and the correlation information of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples includes:

[0030] Denoise the similarity information between every two of the at least two training samples to obtain the denoised similarity information;

[0031] Based on the denoised similarity information, aggregate the multi-modal feature vectors of the first training sample and the multi-modal feature vectors of the second training sample to obtain the denoised multi-modal feature vector of the first training sample;

[0032] Wherein, the first training sample is any one of the at least two training samples, and the second training sample is a training sample whose similarity information with the first training sample is not zero.

[0033] According to one aspect of the present application, a method for training a multi-modal model is provided. Denoising the similarity information between every two of the at least two training samples to obtain denoised similarity information includes:

[0034] When the first similarity is less than the average similarity of the at least two training samples in the first modality, set the first similarity to zero;

[0035] Wherein, the first similarity is used to indicate the similarity between the first training sample and the third training sample in the first modality.

[0036] According to one aspect of the present application, a method for training a multi-modal model is provided. Denoising the similarity information between every two of the at least two training samples to obtain denoised similarity information includes:

[0037] When the second similarity is zero, set the first similarity to zero;

[0038] Wherein, the first similarity is used to indicate the similarity between the first training sample and the third training sample in the first modality, the second similarity is used to indicate the similarity between the first training sample and the third training sample in the second modality, and the second modality is the modality other than the first modality among the at least two modalities.

[0039] According to one aspect of the present application, a method for training a multi-modal model is provided. The denoised multi-modal feature vector of the first training sample includes denoised feature vectors of the first training sample in at least two modalities;

[0040] Based on the denoised similarity information, aggregating the multi-modal feature vector of the first training sample and the multi-modal feature vector of the second training sample to obtain the denoised multi-modal feature vector of the first training sample includes:

[0041] Using the similarity between the first training sample and the second training sample in the i-th modality as a weight, aggregating the feature vectors of the first training sample and the second training sample in the i-th modality to obtain the denoised feature vector of the first training sample in the i-th modality; i is a positive integer;

[0042] Repeating the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first training sample.

[0043] According to one aspect of the present application, there is provided a method for training a multi-modal model, which denoises the multi-modal feature vectors of at least two training samples based on at least one of the similarity information and the correlation information of the at least two training samples to obtain denoised multi-modal feature vectors, including:

[0044] Denoise the correlation information between every two of the at least two training samples to obtain denoised correlation information;

[0045] Based on the denoised correlation information, aggregate the multi-modal feature vectors of the first training sample and the fourth training sample to obtain the denoised multi-modal feature vector of the first training sample;

[0046] Wherein, the fourth training sample is a training sample whose correlation information with the first training sample is not zero.

[0047] According to one aspect of the present application, there is provided a method for training a multi-modal model, which denoises the correlation information between every two of the at least two training samples to obtain denoised correlation information, including:

[0048] In the case where the correlation between the first training sample and the fifth training sample is lower than the correlation threshold, set the correlation between the first training sample and the fifth training sample to zero.

[0049] According to one aspect of the present application, there is provided a method for training a multi-modal model, wherein the denoised multi-modal feature vector of the first training sample includes denoised feature vectors of the first training sample in at least two modalities;

[0050] The aggregating the multi-modal feature vectors of the first training sample and the fourth training sample based on the denoised correlation information to obtain the denoised multi-modal feature vector of the first training sample includes:

[0051] Taking the similarity between the first training sample and the fourth training sample in the i-th modality as a weight, aggregate the feature vectors of the first training sample and the fourth training sample in the i-th modality to obtain the denoised feature vector of the first training sample in the i-th modality; i is a positive integer;

[0052] Repeat the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first training sample.

[0053] According to one aspect of the present application, there is provided a multi-modal recommendation device, the device includes:

[0054] A first acquisition module, configured to acquire multi-modal feature vectors of at least two items to be recommended, where the multi-modal feature vector of each item to be recommended includes feature vectors of each item to be recommended in at least two modalities; and acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended.

[0055] A first denoising module, configured to denoise the multi-modal feature vectors of the at least two items to be recommended to obtain denoised multi-modal feature vectors of the at least two items to be recommended.

[0056] A first calculation module, configured to calculate, based on the denoised multi-modal feature vectors and the object feature vector, the preference scores of the object to be recommended for each of the items to be recommended.

[0057] A first generation module, configured to generate, based on the preference scores of each of the items to be recommended, the recommendation list of the object to be recommended, where the recommendation list includes at least one of the items to be recommended.

[0058] According to one aspect of the present application, there is provided a model training device for a multi-modal model, where the device includes:

[0059] A second acquisition module, configured to acquire multi-modal feature vectors of at least two training samples, where the multi-modal feature vector of each training sample includes feature vectors of the training sample in at least two modalities; and acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object.

[0060] A second denoising module, configured to denoise the multi-modal feature vectors of the at least two training samples to obtain denoised multi-modal feature vectors of the at least two training samples.

[0061] A second calculation module, configured to calculate, based on the denoised multi-modal feature vectors and the object feature vector, the predicted preference scores of the sample object for each of the training samples.

[0062] A first training module, configured to train the multi-modal model based on the sample loss of the training samples.

[0063] According to one aspect of the present application, there is provided a computer device, where the computer device includes: a processor and a memory, where at least one program is stored in the memory; the processor is configured to execute the at least one program in the memory to implement the above multi-modal recommendation method and / or the model training method of the multi-modal model.

[0064] According to one aspect of the present application, there is provided a computer-readable storage medium storing executable instructions, which are loaded and executed by a processor to implement the above multi-modal recommendation method and / or the model training method of the multi-modal model.

[0065] According to one aspect of the present application, there is provided a computer program product including computer instructions stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the above multi-modal recommendation method and / or the model training method of the multi-modal model.

[0066] The beneficial effects brought by the technical solution provided by the present application at least include:

[0067] Denoise the multi-modal feature vector of the item to be recommended, and then generate a preference score for the item to be recommended based on the denoised multi-modal feature vector and the object feature. Among them, the multi-modal feature vector is the feature vector of the item to be recommended in each modality, and the object feature is the preference feature vector of the object to be recommended. Denoising can improve the accuracy of the multi-modal feature vector, thereby improving the accuracy of the calculated preference score, and finally improving the accuracy of the multi-modal recommendation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0069] Figure 1 Shows the architecture diagram of a computer system provided by an exemplary embodiment of the present application;

[0070] Figure 2 Shows the flowchart of a multi-modal recommendation method provided by an exemplary embodiment of the present application;

[0071] Figure 3 Shows the schematic diagram of a multi-modal recommendation method provided by an exemplary embodiment of the present application;

[0072] Figure 4 Shows the schematic diagram of a multi-modal recommendation method provided by an exemplary embodiment of the present application;

[0073] Figure 5 Shows the flowchart of the model training method of a multi-modal model provided by an exemplary embodiment of the present application;

[0074] Figure 6Schematic diagram showing the model training method of a multi-modal model provided by an exemplary embodiment of the present application;

[0075] Figure 7 Schematic diagram showing the model training method of a multi-modal model provided by an exemplary embodiment of the present application;

[0076] Figure 8 Schematic diagram showing the model training method of a multi-modal model provided by an exemplary embodiment of the present application;

[0077] Figure 9 Schematic diagram showing the model training method of a multi-modal model provided by an exemplary embodiment of the present application;

[0078] Figure 10 Block diagram showing the structure of a multi-modal recommendation device provided by an exemplary embodiment of the present application;

[0079] Figure 11 Block diagram showing the structure of a model training device of a multi-modal model provided by an exemplary embodiment of the present application;

[0080] Figure 12 Schematic diagram showing the structure of a computer device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0081] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0082] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0083] The terms used in the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "the" and "said" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0084] It should be noted that the user information involved in this application (including but not limited to the user's feedback information, browsing information, operation information, etc.) and data (including but not limited to data for analysis, training, recommendation, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, the information such as setting operations involved in this application is obtained under full authorization.

[0085] It should be understood that although the terms first, second, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0086] First, introduce the relevant nouns involved in this application:

[0087] Multimodal Recommender System: A recommender system that can process and utilize multiple modalities of data. With the rapid development of multimedia services such as short videos and news, the data to be processed in recommender systems is becoming increasingly diverse, including various modalities of data such as text, images, and audio. Therefore, multimodal recommender systems have emerged and become a research hotspot in the field of current recommender systems. The core idea of multimodal recommender systems is to fuse data of different modalities, so as to more comprehensively understand user needs and item attributes and improve the accuracy and satisfaction of recommendations. Specifically, multimodal recommender systems can use deep learning techniques to extract features and perform representation learning on data of different modalities to obtain their respective vector representations. Then, through some fusion strategies such as weighted average and attention mechanisms, these vector representations are fused to obtain a more comprehensive item representation. Finally, according to the user's historical behavior and preferences, a personalized recommendation list is generated. Thus, the purpose of using multiple modalities of data to more comprehensively understand user needs and item attributes and improve the accuracy and satisfaction of recommendations is achieved.

[0088] The sources of noise in a multi-modal recommendation system are diverse, including noise in multi-modal content, noise in user feedback behavior, etc. Noise in multi-modal content includes misjudgment of the similarity relationship between items. For example, item A and item B are very similar in a certain modality, but in fact, the overall similarity between item A and item B is not high. Noise in multi-modal content also includes misjudgment of the association relationship between items. For example, the association relationship between items is reflected in the number of times a user clicks on two items. That is, when two items appear in the recommendation list at the same time, if a user C clicks on item A and then clicks on item B, or clicks on item B and then clicks on item A, it is considered that there is a certain association between item A and item B. The feedback behavior of users is the reflection of users on the recommendation results. The feedback behavior of users includes various forms, such as clicks, purchases, comments, ratings, etc. The noise in the feedback behavior of users is manifested as the feedback given by users to items being contrary to the actual preferences of users. For example, a user is attracted by the information of an item in a certain modality, but the user actually does not like this item; or, a user accidentally clicks the like button for an item, but cancels the like later.

[0089] Figure 1 FIG. shows a block diagram of the structure of a computer system 100 provided by an exemplary embodiment of the present application. The computer system 100 includes: a server 110 and a terminal 120.

[0090] The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services, such as a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and cloud servers that provide basic cloud computing services such as big data and artificial intelligence platforms. The server 110 is used to calculate the preference score of the object to be recommended based on the item features and object features; the server 110 is also used to generate a recommendation list for the object to be recommended based on the calculated preference score of the object to be recommended.

[0091] The terminal 120 can be an electronic device such as a mobile phone, a tablet computer, an in-vehicle terminal (car computer), a wearable device, a PC (Personal Computer), an unattended reservation terminal, a smart speaker, etc. The terminal 120 can be a terminal that installs and runs a target application. The target application can be an application that supports multi-modal recommendation, such as an online video playback application, a shopping application, a social application, etc. The form of the target application in the embodiments of the present application is not limited, including but not limited to a client, an application, a small program, etc. installed in the terminal 120, and can also be in the form of a web page. The terminal 120 is used to obtain the feedback behavior of the user on the recommendation list.

[0092] Communication can be carried out between the server 110 and the terminal 120 via a network, such as a wired or wireless network.

[0093] Those skilled in the art can be aware that the number of the above-mentioned terminals 120 can be more or less. For example: the above-mentioned terminal 120 can be only one, or dozens or hundreds, or a larger number. The embodiments of the present application do not limit the number and device type of the terminal 120.

[0094] Figure 2 The flowchart of a multi-modal recommendation method provided by an exemplary embodiment of the present application is shown. The method is executed by a server or a terminal, and the server or terminal can be Figure 1 the server or terminal shown. The method includes the following steps.

[0095] Step 210: Obtain multi-modal feature vectors of at least two items to be recommended. Each multi-modal feature vector of an item to be recommended includes feature vectors of each item to be recommended in at least two modalities; and obtain an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended.

[0096] In some embodiments, for different application scenarios of the multi-modal recommendation method, the items to be recommended and the objects to be recommended are different. Exemplarily, the items to be recommended include at least one of commodities, articles, videos, advertisements; the objects to be recommended include at least one of users, public display screens. For example, when the multi-modal recommendation method is applied in an e-commerce application program, the item to be recommended is a commodity and the object to be recommended is a user; when the multi-modal recommendation method is applied in a short video application program, the items to be recommended are videos or articles and the object to be recommended is a user; when the multi-modal recommendation method is applied in the placement of public advertisements, the item to be recommended is an advertisement and the object to be recommended is a public display screen, such as an electronic screen for playing advertisements.

[0097] In some embodiments, based on a multi-modal model, multi-modal feature vectors of at least two items to be recommended are extracted.

[0098] Step 220: Denoise the multi-modal feature vectors of at least two items to be recommended to obtain denoised multi-modal feature vectors of at least two items to be recommended.

[0099] In some embodiments, the multi-modal feature vector of each item to be recommended includes the feature vectors of each item to be recommended in at least two modalities. The denoised multi-modal feature vector of each item to be recommended includes the denoised feature vectors of each item to be recommended in at least two modalities. Denoising the multi-modal feature vectors of at least two items to be recommended to obtain the denoised multi-modal feature vectors of at least two items to be recommended can be understood as denoising the feature vectors of each item to be recommended in at least two modalities to obtain the denoised feature vectors of each item to be recommended in at least two modalities.

[0100] Step 230: Calculate the preference scores of the object to be recommended for each item to be recommended based on the denoised multi-modal feature vector and the object feature vector.

[0101] In some embodiments, the denoised multi-modal feature vector of each item to be recommended includes the denoised feature vectors of each item to be recommended in at least two modalities. Calculate the preference scores of the object to be recommended for each item to be recommended in at least two modalities based on the denoised feature vectors of each object to be recommended in at least two modalities and the object features.

[0102] Exemplarily, the multi-modal feature vector includes a visual modal feature vector, a text modal feature vector, and a behavior modal feature vector. Calculate the preference scores of the object to be recommended for each item to be recommended in the visual modality based on the denoised feature vector of each object to be recommended in the visual modality and the object features; calculate the preference scores of the object to be recommended for each item to be recommended in the text modality based on the denoised feature vector of each object to be recommended in the text modality and the object features; calculate the preference scores of the object to be recommended for each item to be recommended in the behavior modality based on the denoised feature vector of each object to be recommended in the behavior modality and the object features.

[0103] In some embodiments, calculate the preference scores of the object to be recommended for each item to be recommended based on the preference scores of the object to be recommended for each item to be recommended in at least two modalities.

[0104] In some embodiments, aggregate the denoised feature vectors of at least two modalities in the denoised multi-modal feature vector of each item to be recommended to obtain the denoised total feature vector of each item to be recommended; calculate the preference scores of the object to be recommended for each item to be recommended based on the denoised total feature vector of each item to be recommended and the object feature vector.

[0105] Step 240: Generate a recommendation list for the object to be recommended based on the preference scores of each item to be recommended, where the recommendation list includes at least one item to be recommended.

[0106] In some embodiments, based on the preference scores of each item to be recommended, the items to be recommended with the top n preference scores are obtained, and a recommendation list for the object to be recommended is generated. Optionally, in the recommendation list, at least one item to be recommended is arranged in descending order of preference score.

[0107] In summary, the method provided by the embodiments of the present application denoises the multi-modal feature vectors of the items to be recommended, and then generates the preference scores of the items to be recommended according to the denoised multi-modal feature vectors and the object features. Among them, the multi-modal feature vectors are the feature vectors of the items to be recommended in each modality, and the object features are the preference feature vectors of the object to be recommended. Denoising can improve the accuracy of the multi-modal feature vectors, thereby improving the accuracy of the calculated preference scores, and finally improving the accuracy of the multi-modal recommendation results.

[0108] In the optional embodiment based on Figure 2 Step 220 may be alternatively implemented as Step 221.

[0109] Step 221: Denoise the multi-modal feature vectors of at least two items to be recommended based on at least one of the similarity information and the correlation information of the at least two items to be recommended, and obtain the denoised multi-modal feature vectors of the at least two items to be recommended.

[0110] Among them, the similarity information is used to indicate the similarity degree between two items to be recommended in the same modality, and the correlation information is used to indicate the correlation degree between two items to be recommended.

[0111] In some embodiments, the similarity information is determined based on at least one of the content of the item to be recommended and the behavior of the object. The content of the item to be recommended includes the similarity in each modality of the item to be recommended, such as the similarity of text, the similarity of pictures, etc. The similarity of the content of two items to be recommended can be calculated through the feature vectors of the item to be recommended in the corresponding modality. The object behavior includes the historical behavior of the object to be recommended towards the item to be recommended, such as historical ratings, historical clicks, etc. The object to be recommended can be the object of this recommendation or an object with similar preferences to the object of this recommendation.

[0112] In some embodiments, the correlation information is determined based on at least one of the content of the item to be recommended and the behavior of the object. The content of the item to be recommended includes the correlation in each modality of the item to be recommended, such as the correlation of text, the correlation of pictures, etc. The correlation of the content of two items to be recommended can be calculated through the feature vectors of the item to be recommended in the corresponding modality. The object behavior includes the historical behavior of the object to be recommended towards the item to be recommended, such as the object to be recommended purchasing different items to be recommended at the same time, the object to be recommended browsing different items to be recommended in the recommendation list successively, etc. The object to be recommended can be the object of this recommendation or an object with similar preferences to the object of this recommendation.

[0113] In summary, the method provided by the embodiments of the present application performs denoising based on the similarity information and / or correlation information of at least two items to be recommended. Since the similarity information and correlation information are important attributes for multimodal recommendation, using the similarity information and / or correlation information for denoising can improve the accuracy of the multimodal recommendation method.

[0114] Next, two ways of denoising the multimodal feature vectors of at least two items to be recommended based on at least one of the similarity information and correlation information of at least two items to be recommended to obtain the denoised multimodal feature vectors of at least two items to be recommended are shown. The order of introduction does not represent the superiority or inferiority of the ways.

[0115] Way 1: Denoising based on similarity information.

[0116] Way 2: Denoising based on correlation information.

[0117] Among them, Way 1 and Way 2 can be combined and implemented, that is, denoising based on similarity information and correlation information. The embodiments of the present application do not limit this.

[0118] Way 1: Denoising based on similarity information.

[0119] In an optional embodiment, step 221 can be alternatively implemented as step 310 and step 320.

[0120] Step 310: Denoise the similarity information between every two items to be recommended among at least two items to be recommended to obtain the denoised similarity information.

[0121] In some embodiments, the similarity information between every two items to be recommended is calculated based on the multimodal feature vectors of every two items to be recommended. The calculation methods of similarity information include at least one of cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, and Jaccard similarity.

[0122] In some embodiments, the similarity information is denoised. There are many sources of noise in the similarity information, including inaccurate calculation of the similarity information due to less interaction information between the item to be recommended and the object to be recommended, and the existence of similarity differences between different modalities due to different evaluation criteria for similarity between different modalities.

[0123] In some embodiments, the similarity information is used to indicate the similarity degree of two items to be recommended in the same modality.

[0124] In an optional embodiment, step 310 can be alternatively implemented as at least one of step 311 and step 312.

[0125] Step 311: When the first similarity is less than the average similarity of at least two items to be recommended in the first modality, set the first similarity to zero.

[0126] Wherein, the first similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the first modality.

[0127] In some embodiments, calculate the average similarity of at least two items to be recommended in each modality. When the similarity between two items to be recommended in the k-th modality is less than the average similarity corresponding to the k-th modality, it is considered that the similarity between the two items to be recommended in the k-th modality is noise, and set the similarity between the two items to be recommended in the k-th modality to zero.

[0128] In some embodiments, a similarity of zero means that two items to be recommended are not similar, and the feature vectors of two items to be recommended with a similarity of zero will not be aggregated when performing feature aggregation.

[0129] Step 312: When the second similarity is zero, set the first similarity to zero.

[0130] Wherein, the first similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the first modality, and the second similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the second modality. The second modality is a modality other than the first modality among at least two modalities.

[0131] In some embodiments, when the similarity between two items to be recommended in other modalities is zero, set the similarity between the two items to be recommended in the current modality to zero.

[0132] In some embodiments, Step 311 and Step 312 can be executed in reverse order or simultaneously. Step 311 can be executed alone to denoise the similarity information, Step 312 can be executed alone to denoise the similarity information, and Step 311 and Step 312 can also be executed in combination to denoise the similarity information.

[0133] Exemplarily, as Figure 3 shown, there are 6 items to be recommended, namely a1 - a6, and at least two modalities include the visual modality and the text modality. The connection lines between the items to be recommended indicate that the similarity information between two items to be recommended is not zero. From Figure 3It can be obtained that the similarity information between the item to be recommended a1 and the item to be recommended a2 in the visual modality is 0.8; the similarity information between the item to be recommended a2 and the item to be recommended a3 is 0.8; the similarity information between the item to be recommended a3 and the item to be recommended a4 is 0.5; the similarity information between the item to be recommended a3 and the item to be recommended a5 is 0.2; the similarity information between the item to be recommended a5 and the item to be recommended a6 is 0.1; the similarity information between the item to be recommended a6 and the item to be recommended a1 is 0.6. In the text modality, the similarity information between the item to be recommended a1 and the item to be recommended a2 is 0.6; the similarity information between the item to be recommended a2 and the item to be recommended a3 is 0.8; the similarity information between the item to be recommended a3 and the item to be recommended a4 is 0.1; the similarity information between the item to be recommended a3 and the item to be recommended a5 is 0.4; the similarity information between the item to be recommended a6 and the item to be recommended a2 is 0.7.

[0134] Exemplarily, the method corresponding to step 311 is adopted for similarity denoising. First, calculate the average similarity in the visual modality, which is (0.8 + 0.8 + 0.5 + 0.2 + 0.1 + 0.6) / 6, and the average similarity is obtained as 0.5. Therefore, set the similarity between the item to be recommended a3 and the item to be recommended a5 in the visual modality to zero, and set the similarity between the item to be recommended a5 and the item to be recommended a6 in the visual modality to zero. Then calculate the average similarity in the text modality, which is (0.6 + 0.8 + 0.7 + 0.1 + 0.4) / 6, and the average similarity is approximately 0.43. Therefore, set the similarity between the item to be recommended a2 and the item to be recommended a2 in the text modality to zero; set the similarity between the item to be recommended a3 and the item to be recommended a5 to zero.

[0135] Exemplarily, the method corresponding to step 312 is adopted for similarity denoising. The connection line between the items to be recommended indicates that the similarity information between two items to be recommended is not zero. Therefore, the similarity between the item to be recommended a2 and the item to be recommended a6 in the visual modality is zero, and set the similarity between the item to be recommended a2 and the item to be recommended a6 in the text modality to zero. The similarity between the item to be recommended a1 and the item to be recommended a6 in the text modality is zero, and set the similarity between the item to be recommended a1 and the item to be recommended a6 in the visual modality to zero.

[0136] Exemplarily, the methods corresponding to step 311 and step 312 are adopted for similarity denoising, and the final similarity connection relationship between the items to be recommended is as Figure 3 shown.

[0137] Step 320: Based on the denoised similarity information, aggregate the multi-modal feature vectors of the first item to be recommended and the multi-modal feature vectors of the second item to be recommended to obtain the denoised multi-modal feature vectors of the first item to be recommended.

[0138] Wherein, the first item to be recommended is any one of the at least two items to be recommended, and the second item to be recommended is an item to be recommended whose similarity information with the first item to be recommended is not zero.

[0139] In some embodiments, based on the denoised similarity information, a denoised multi-modal feature vector of the first item to be recommended is aggregated. When aggregating the denoised multi-modal feature vector of the first item to be recommended, the multi-modal feature vector of the first item to be recommended and the multi-modal feature vectors of other items to be recommended whose similarity information with the first item to be recommended is not zero are aggregated.

[0140] Exemplarily, there are 5 items to be recommended, namely a1 - a5. After similarity denoising, the items to be recommended whose similarity information with the item to be recommended a1 is not zero are a2 and a4. When aggregating the item to be recommended a1 based on the denoised similarity information, the multi-modal feature vectors of the item to be recommended a1, a2, and a4 are aggregated to obtain the denoised multi-modal feature vector of the item to be recommended a1.

[0141] In some embodiments, the similarity information is used to indicate the similarity degree between two items to be recommended in the same modality. Based on the denoised similarity information in the i-th modality, the feature vector of the first item to be recommended in the k-th modality and the feature vector of the second item to be recommended in the i-th modality are aggregated to obtain the denoised feature vector of the first item to be recommended in the i-th modality, where the second item to be recommended is an item to be recommended whose similarity information with the first item to be recommended is not zero, and i is a positive integer.

[0142] In an alternative embodiment, step 320 can be alternatively implemented as steps 321 and 322.

[0143] Step 321: Using the similarity between the first item to be recommended and the second item to be recommended in the i-th modality as a weight, aggregate the feature vectors of the first item to be recommended and the second item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer.

[0144] In some embodiments, the denoised multi-modal feature vector of the first item to be recommended includes denoised feature vectors of the first item to be recommended in at least two modalities.

[0145] In some embodiments, for the first item to be recommended, using the similarity between the first item to be recommended and the second item to be recommended in the k-th modality as a weight, or calculating a weight according to the similarity between the first item to be recommended and the second item to be recommended in the i-th modality, aggregate the feature vectors of the first item to be recommended and the second item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality.

[0146] Exemplarily, there are 5 items to be recommended, namely a1 - a5, and the corresponding feature vectors of the items to be recommended are denoted as A1 - A5 respectively. After similarity denoising, the items to be recommended with non - zero similarity information to the item to be recommended a1 are a2 and a4, and their corresponding similarities are 0.8 and 0.3 respectively. Optionally, directly use the similarities corresponding to the items to be recommended a2 and a4 as weights. For example, the denoised feature vector corresponding to the item to be recommended a1 is A1 + 0.8A2 + 0.3A4; or, calculate the weights according to the similarity between the first item to be recommended and the second item to be recommended in the i - th modality. For example, the weight corresponding to the item to be recommended a2 is 0.8 / (0.8 + 0.3), that is, 8 / 11, and the weight corresponding to the item to be recommended a4 is 0.3 / (0.8 + 0.3), that is, 3 / 11. The denoised feature vector corresponding to the item to be recommended a1 is A1+(8 / 11)A2+(3 / 11)A4. It should be noted that for the aggregation result of the feature vectors, the embodiments of the present application only take simple addition as an example for illustration, but do not limit the aggregation method of the feature vectors.

[0147] Step 322: Repeat the above step for other modalities different from the i - th modality to obtain the denoised multi - modal feature vector of the first item to be recommended.

[0148] In some embodiments, the denoised multi - modal feature vector of the first item to be recommended can be aggregated by the above method, ensuring the execution of the multi - modal recommendation method and the accuracy of the multi - modal recommendation method.

[0149] In summary, the method provided by the embodiments of the present application first denoises the similarity information between two items to be recommended, and then aggregates the feature vectors according to the denoised similarity information to obtain the denoised feature vector. Similarity information is one of the important attributes of the multi - modal recommendation method. By denoising the feature vectors through similarity information, the accuracy of the multi - modal recommendation method can be improved well. And, the embodiments of the present application also show two methods for denoising similarity information. These two methods can be implemented independently or in combination. The first denoising method considers that the similarity relationship with similarity lower than the average similarity belongs to noise, and the second denoising method considers that when it is dissimilar in other modalities, it should also be dissimilar in the current modality; these two denoising methods denoise the similarity from different angles, which can improve the accuracy of similarity denoising and thus improve the accuracy of the multi - modal recommendation method.

[0150] Method 2: Denoising based on correlation information.

[0151] In an alternative embodiment, step 221 can be alternatively implemented as steps 330 and 340.

[0152] Step 330: Denoise the correlation information between each pair of at least two items to be recommended to obtain the denoised correlation information.

[0153] In some embodiments, the relevance information is used to indicate the degree of relevance between two items to be recommended. Optionally, the relevance information between every two items to be recommended is the number of interactions of a certain recommendation object with respect to the two items to be recommended; or, the relevance information between every two items to be recommended is the interaction frequency of a certain recommendation object with respect to the two items to be recommended; or, the relevance information between every two items to be recommended is the interaction score of a certain recommendation object with respect to the two items to be recommended, and the interaction score is calculated based on the number of interactions and the weights corresponding to the interactions.

[0154] In some embodiments, the relevance information between every two items to be recommended is the number of interactions of a certain recommendation object with respect to the two items to be recommended. Optionally, the relevance information includes the relevance information of different interactions. For example, for a click operation, there is a click relevance; or, for a like operation, there is a like relevance; or, for a browse operation, there is a browse relevance.

[0155] Exemplarily, there are 5 items to be recommended, namely a1 - a5, and the recommendation object is a user. There are 2 users, namely b1 and b2. When the items to be recommended a1 - a5 appear simultaneously, user b1 first interacts with item to be recommended a1, and then immediately interacts with item to be recommended a3. At this time, the relevance between item to be recommended a1 and a3 is incremented by 1; user b2 first interacts with item to be recommended a3, and then interacts with item to be recommended a1, then the relevance between item to be recommended a1 and a3 is incremented by 1 again.

[0156] In some embodiments, the noise in the relevance information comes from the mis - interaction of the recommendation object with respect to the item to be recommended, and this noise can be determined by the number of interactions, that is, the value of the relevance.

[0157] In an alternative embodiment, step 330 can be alternatively implemented as step 331.

[0158] Step 331: When the relevance between the first item to be recommended and the fifth item to be recommended is lower than the relevance threshold, set the relevance between the first item to be recommended and the fifth item to be recommended to zero.

[0159] In some embodiments, when the relevance between the first item to be recommended and the fifth item to be recommended is lower than the relevance threshold, the relevance between the first item to be recommended and the fifth item to be recommended may be noise. Therefore, set the relevance between the first item to be recommended and the fifth item to be recommended to zero.

[0160] Exemplarily, such as Figure 4As shown, there are 6 items to be recommended, namely a1 - a6. The connecting lines between the items to be recommended indicate that the correlation information between two items to be recommended is not zero. The correlation between item to be recommended a1 and item to be recommended a2 is 1; the correlation between item to be recommended a2 and item to be recommended a3 is 10; the correlation between item to be recommended a2 and item to be recommended a5 is 6; the correlation between item to be recommended a5 and item to be recommended a1 is 8; the correlation between item to be recommended a1 and item to be recommended a4 is 1; the correlation between item to be recommended a1 and item to be recommended a6 is 2; the correlation between item to be recommended a6 and item to be recommended a2 is 15. When the correlation threshold is 2, the correlation between item to be recommended a1 and item to be recommended a2 is lower than 2, and the correlation between item to be recommended a1 and item to be recommended a2 is set to zero; the correlation between item to be recommended a1 and item to be recommended a4 is lower than 2, and the correlation between item to be recommended a1 and item to be recommended a2 is set to zero.

[0161] Step 340: Based on the denoised correlation information, aggregate the multi-modal feature vectors of the first item to be recommended and the fourth item to be recommended to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0162] Wherein, the fourth item to be recommended is the item to be recommended whose correlation information with the first item to be recommended is not zero.

[0163] In some embodiments, based on the denoised correlation information, the denoised multi-modal feature vector of the first item to be recommended is aggregated. When aggregating the denoised multi-modal feature vector of the first item to be recommended, it is by aggregating the multi-modal feature vector of the first item to be recommended and the multi-modal feature vectors of other items to be recommended whose correlation information with the first item to be recommended is not zero.

[0164] Exemplarily, there are 5 items to be recommended, namely a1 - a5. After correlation denoising, the items to be recommended whose correlation information with item to be recommended a1 is not zero are a2 and a4. When aggregating item to be recommended a1 based on the denoised correlation information, aggregate the multi-modal feature vectors of item to be recommended a1, a2, and a4 to obtain the denoised multi-modal feature vector of item to be recommended a1.

[0165] In some embodiments, the correlation information is used to indicate the correlation degree between two items to be recommended. Based on the denoised correlation information, aggregate the feature vector of the first item to be recommended in the i-th modality and the feature vector of the fourth item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality, where the fourth item to be recommended is the item to be recommended whose correlation information with the first item to be recommended is not zero.

[0166] In an alternative embodiment, step 340 can be alternatively implemented as steps 341 and 342.

[0167] Step 341: Using the similarity between the first item to be recommended and the fourth item to be recommended in the i-th modality as the weight, aggregate the feature vectors of the first item to be recommended and the fourth item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer.

[0168] In some embodiments, the denoised multi-modal feature vector of the first item to be recommended includes the denoised feature vectors of the first item to be recommended in at least two modalities.

[0169] In some embodiments, for the first item to be recommended, using the correlation degree between the first item to be recommended and the fourth item to be recommended in the i-th modality as the weight, or calculating the weight according to the correlation degree between the first item to be recommended and the fourth item to be recommended in the i-th modality, aggregate the feature vectors of the first item to be recommended and the fourth item to be recommended in the i-th modality to obtain the denoised feature vector of the first item to be recommended in the i-th modality.

[0170] Exemplarily, there are 5 items to be recommended, namely a1 - a5, and the corresponding feature vectors of the items to be recommended are denoted as A1 - A5 respectively. After denoising by correlation degree, the items to be recommended with non-zero correlation degree information with the item to be recommended a1 are a2 and a4, and their corresponding correlation degrees are 0.8 and 0.3 respectively. Optionally, directly using the correlation degrees corresponding to the items to be recommended a2 and a4 as the weights, for example, the denoised feature vector corresponding to the item to be recommended a1 is A1 + 0.8A2 + 0.3A4; or calculating the weights according to the correlation degrees between the first item to be recommended and the fourth item to be recommended, for example, the weight corresponding to the item to be recommended a2 is 0.8 / (0.8 + 0.3), that is, 8 / 11, the weight corresponding to the item to be recommended a4 is 0.3 / (0.8 + 0.3), that is, 3 / 11, and the denoised feature vector corresponding to the item to be recommended a1 is A1 + (8 / 11)A2 + (3 / 11)A4. It should be noted that for the aggregation result of the feature vectors, the embodiments of the present application only use simple addition as an example for illustration, but do not limit the aggregation method of the feature vectors.

[0171] Step 342: Repeat the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0172] In summary, the method provided by the embodiments of the present application first denoises the correlation information between two items to be recommended, and then aggregates the feature vectors according to the denoised correlation information to obtain denoised feature vectors. The correlation information is one of the important attributes of the multi-modal recommendation method. By denoising the feature vectors through the correlation information, the accuracy of the multi-modal recommendation method can be improved well. When denoising the correlation information, it is considered that the correlation between two items to be recommended below the correlation threshold is noise, and then it is set to zero to remove the influence of the noise on the subsequent multi-modal recommendation method, which can improve the accuracy of the multi-modal recommendation method.

[0173] Figure 5 FIG. 4 shows a flowchart of a model training method for a multi-modal model provided by an exemplary embodiment of the present application. The method is executed by a server, and the server can be Figure 1 the server shown. The method includes the following steps.

[0174] Step 410: Obtain multi-modal feature vectors of at least two training samples. The multi-modal feature vector of each training sample includes feature vectors of the training sample in at least two modalities; and obtain an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object.

[0175] In some embodiments, for different application scenarios of the multi-modal recommendation method, the training samples and the objects to be recommended are different. Exemplarily, the training samples include at least one of commodities, articles, videos, and advertisements; the objects to be recommended include at least one of users and public display screens. For example, when the multi-modal recommendation method is applied in an e-commerce application program, the training samples are commodities and the objects to be recommended are users; when the multi-modal recommendation method is applied in a short video application program, the training samples are videos or articles and the objects to be recommended are users; when the multi-modal recommendation method is applied to the placement of public advertisements, the training samples are advertisements and the objects to be recommended are public display screens, such as electronic screens for playing advertisements.

[0176] Step 420: Denoise the multi-modal feature vectors of at least two training samples to obtain denoised multi-modal feature vectors of at least two training samples.

[0177] In some embodiments, the multi-modal feature vector of each item to be recommended includes feature vectors of each training sample in at least two modalities. The denoised multi-modal feature vector of each training sample includes denoised feature vectors of each training sample in at least two modalities. Denoising the multi-modal feature vectors of at least two training samples to obtain denoised multi-modal feature vectors of at least two training samples can be understood as denoising the feature vectors of each training sample in at least two modalities to obtain denoised feature vectors of each training sample in at least two modalities.

[0178] Step 430: Calculate the preference score of the sample object for each training sample based on the denoised multi-modal feature vector and the object feature vector.

[0179] In some embodiments, the denoised multi-modal feature vector of each training sample includes the denoised feature vectors of each training sample in at least two modalities. Based on the denoised feature vectors of each object to be recommended in at least two modalities and the object features, calculate the preference scores of the object to be recommended for each training sample in at least two modalities.

[0180] Exemplarily, the multi-modal feature vector includes a visual modality feature vector, a text modality feature vector, and a behavior modality feature vector. Based on the denoised feature vector of each object to be recommended in the visual modality and the object features, calculate the preference score of the object to be recommended for each training sample in the visual modality; based on the denoised feature vector of each object to be recommended in the text modality and the object features, calculate the preference score of the object to be recommended for each training sample in the text modality; based on the denoised feature vector of each object to be recommended in the behavior modality and the object features, calculate the preference score of the object to be recommended for each training sample in the behavior modality.

[0181] In some embodiments, calculate the preference score of the object to be recommended for each training sample based on the preference scores of the object to be recommended for each training sample in at least two modalities.

[0182] In some embodiments, aggregate the denoised feature vectors of at least two modalities in the denoised multi-modal feature vector of each training sample to obtain the denoised total feature vector of each training sample; based on the denoised total feature vector of each training sample and the object feature vector, calculate the preference score of the object to be recommended for each training sample.

[0183] Step 440: Train the multi-modal model based on the sample loss of the training samples.

[0184] In summary, the method provided by the embodiments of the present application shows a model training method for a multi-modal model. When training the multi-modal model, it will first denoise the multi-modal feature vector of the training samples and then train based on the denoised multi-modal feature vector. This method enables the multi-modal model to learn the denoised multi-modal feature vector, enables the multi-modal model to more accurately learn the features of the training samples, and thus improves the accuracy of the multi-modal model.

[0185] In some embodiments, the sample loss includes at least one of the following:

[0186] · The first loss, which is used to indicate the loss between the ranking of the predicted preference scores of the sample object for each training sample and the true preference ranking of the sample object for each training sample, and the true preference ranking is used to indicate the preference degree of the sample object for each training sample;

[0187] · The second loss is used to indicate the loss between the multimodal preference score calculated based on the multimodal feature vectors of the training samples and the collaborative preference score calculated based on the collaborative feature vectors of the training samples;

[0188] · The third loss is used to indicate the loss of contrastive learning with the multimodal similarity set as the positive sample and the unimodal similarity set and the dissimilarity set as the negative samples;

[0189] · The fourth loss is used to indicate the loss of contrastive learning with the unimodal similarity set as the positive sample and the dissimilarity set as the negative sample.

[0190] It should be noted that the sample loss includes at least one of the first loss, the second loss, the third loss, and the fourth loss. Based on different selections of the sample loss, the training direction of the multimodal model is also different. The above four sample losses can be combined in pairs, in threes, and all together. For example, the sample loss of the multimodal model includes the first loss and the second loss; or, the sample loss of the multimodal model includes the first loss and the third loss; or, the sample loss of the multimodal model includes the first loss and the fourth loss; or, the sample loss of the multimodal model includes the second loss and the third loss; or, the sample loss of the multimodal model includes the second loss and the fourth loss; or, the sample loss of the multimodal model includes the third loss and the fourth loss; or, the sample loss of the multimodal model includes the first loss, the second loss, and the third loss; or, the sample loss of the multimodal model includes the first loss, the second loss, and the fourth loss; or, the sample loss of the multimodal model includes the second loss, the third loss, and the fourth loss; or, the sample loss of the multimodal model includes the first loss, the second loss, the third loss, and the fourth loss. The specific content of the sample loss in the embodiments of the present application is not limited.

[0191] Next, the calculation methods of the above four sample losses are introduced.

[0192] · Calculation of the first loss

[0193] Step 510: Classify at least two training samples based on the feedback behaviors of the sample object on at least one training sample to obtain positive samples and negative samples. The positive samples are the training samples with feedback behaviors, and the negative samples are the training samples without feedback behaviors.

[0194] In some embodiments, based on the feedback behaviors of the sample object on at least one training sample, the true preference score of the sample object on at least one training sample is calculated. The feedback behavior is used to indicate the behavior of the sample object towards the training sample.

[0195] In some embodiments, the positive samples are training samples with feedback behavior; the negative samples are training samples without feedback behavior. Optionally, the positive samples include noise samples; and / or, the negative samples include noise samples.

[0196] In some embodiments, the negative samples are obtained by sampling.

[0197] Step 520: Obtain an observation triple for training, where the observation triple includes a sample object, a positive sample, and a negative sample. The observation triple is used to indicate that the sample object has a greater preference for the positive sample than for the negative sample. The true preference is used to indicate that the predicted preference score of the positive sample is greater than that of the negative sample, and the noise preference is used to indicate that the predicted preference score of the positive sample is less than that of the negative sample.

[0198] In some embodiments, when calculating the first loss, the input of the multi-modal model exists in the form of an observation triple. The first data of the observation triple is the sample object, the second data is the positive sample, and the third data is the negative sample. This observation triple indicates that the sample object has a greater preference for the positive sample than for the negative sample. Optionally, the observation triple can be expressed as <sample object, positive sample, negative sample>.

[0199] Step 530: Calculate the probability that the observation triple is predicted as a true preference and the probability that the observation triple is predicted as a noise preference.

[0200] In some embodiments, calculate the probability that the observation triple is predicted as a true preference and the probability that the observation triple is predicted as a noise preference. The observation triple being predicted as a noise preference means that the true preference of the sample object is opposite to the preference shown by the observation triple, that is, the preference score of the negative sample is greater than that of the positive sample, that is, the sample object has a greater preference for the negative sample than for the positive sample.

[0201] In some embodiments, based on the predicted preference scores of the training samples, calculate the probability that the observation triple is predicted as a true preference and the probability that the observation triple is predicted as a noise preference respectively.

[0202] The formula for the probability that the observation triple is predicted as a true preference is as follows.

[0203]

[0204] In the formula, μ ui represents the expectation of the average preference score of the sample object u for the training sample i in each modality; represents the variance of the preference score of the sample object u for the training sample i in each modality; α and β are hyperparameters.

[0205] Among them, the expectation μ ui is calculated as shown in the following formula.

[0206]

[0207] In the formula, m represents modality M; M represents all modalities of training sample i; represents the preference score of sample object u for training sample i on modality m; |M| represents the number of modalities.

[0208] Variance is calculated as shown in the following formula.

[0209]

[0210] In the formula, m represents modality M; M represents all modalities of training sample i; represents the preference score of sample object u for training sample i on modality m; |M| represents the number of modalities; μ ui represents the expectation of the average preference score of sample object u for training sample i on each modality.

[0211] The predicted preference score of sample object u for training sample i on modality m is calculated as shown in the following formula.

[0212]

[0213] In the formula, u u represents the object feature vector of sample object u; (u u ) T represents the transpose of the object feature vector of sample object u; represents the feature vector of training sample i on modality m.

[0214] The probability formula for predicting the observed triple as a noisy preference is as follows.

[0215]

[0216] In the formula, represents the maximum preference score of sample object u on training sample i in any modality; represents the average preference score of training samples in the current training batch (batch); γ is a hyperparameter; μ ui represents the expectation of the average preference score of sample object u for training sample i on each modality.

[0217] Among them, is calculated as follows.

[0218]

[0219] In the formula, B represents the current training batch; represents the predicted preference score of the sample object u for the training sample j; |B| represents the number of training samples in the current training batch.

[0220] In some embodiments, the training of the multi-modal model based on the first loss is performed in training batches, and one training batch includes at least one observed triple.

[0221] Step 540: Calculate the first loss. The first loss is the sum of the first product value and the second product value. The first product value is the product of the probability that the observed triple is predicted as a true preference and the first difference. The second product value is the product of the probability that the observed triple is predicted as a noise preference and the second difference. The first difference is the difference between the predicted preference score of the positive sample and the predicted preference score of the negative sample. The second difference is the difference between the predicted preference score of the negative sample and the predicted preference score of the positive sample.

[0222] In some embodiments, the loss function corresponding to the first loss is as follows.

[0223]

[0224] In the formula, F(U, I) represents the probability that the observed triple is predicted as a true preference; represents the predicted preference score of the sample object u for the training sample i, that is, the predicted preference score of the positive sample; represents the predicted preference score of the sample object u for the training sample j, that is, the predicted preference score of the negative sample; g(u, i) represents the probability that the observed triple is predicted as a noise preference; θ represents the model parameters of the multi-modal model.

[0225] Exemplarily, as Figure 6 shown, based on the true value and the predicted value in the user's feedback behavior for model training, the probability that the observed triple is predicted as a true preference and the probability that the observed triple is predicted as a noise preference in the loss function are calculated based on the multi-modal feature vector.

[0226] In summary, the method provided by the embodiments of the present application trains the multi-modal model through the process of combining the true preference score of the user's feedback behavior, that is, the user's true preference score for the training sample, with the predicted preference score predicted by the multi-modal model. The training samples include positive samples and negative samples. When training, it is necessary to combine positive and negative samples to calculate the probability that the observed triple belongs to the noise sample. The model trained in this way can identify the noise sample, and thus achieve the purpose of denoising the user's feedback behavior by the multi-modal model, thereby improving the accuracy of the multi-modal model.

[0227] ·Calculation of the second loss

[0228] Step 610: Obtain the collaborative feature vectors of at least two training samples, where the collaborative feature vectors are used to indicate the feature vectors of the sample object's feedback behavior on the at least two training samples.

[0229] In some embodiments, the collaborative filtering model is used for the feedback behavior of the sample object to obtain the collaborative feature vectors of at least two training samples and the object feature vector of the sample object. Optionally, the collaborative filtering model is part of a multi-modal model; or, the collaborative filtering model and the multi-modal model are two different models.

[0230] In some embodiments, the feedback behavior is used to indicate the behavior of the sample object towards the training sample.

[0231] Step 620: Calculate the collaborative preference score of the sample object for each training sample based on the collaborative feature vectors of the at least two training samples and the object feature vector of the sample object.

[0232] In some embodiments, the collaborative preference score is calculated as follows.

[0233]

[0234] In the formula, u u is the object feature vector of the sample object u; is the collaborative feature vector of the training sample i; represents the collaborative preference score of the sample object u for the training sample i.

[0235] Step 630: Calculate the multi-modal preference score of the sample object for each training sample based on the multi-modal feature vectors of the at least two training samples and the object feature vector of the sample object.

[0236] In some embodiments, the multi-modal preference score is calculated as follows.

[0237]

[0238] In the formula, u u is the object feature vector of the sample object u; is the multi-modal fusion feature vector of the training sample i; represents the multi-modal preference score of the sample object u for the training sample i.

[0239] The multi-modal fusion feature vector is obtained by fusing the feature vectors of at least two modalities in the multi-modal feature vectors, and the fusion formula is as follows.

[0240]

[0241] In the formula, Denote the feature vector of training sample i on modalities a1, a2, …, an; Meanpooling represents the average pooling method for fusing feature vectors of different modalities.

[0242] Step 640: Calculate the first KL divergence and the second KL divergence. The first KL divergence is the KL divergence calculated with the distribution of collaborative preference scores as the original probability distribution and the distribution of multi-modal preference scores as the approximate probability distribution; the second KL divergence is the KL divergence calculated with the distribution of multi-modal preference scores as the original probability distribution and the distribution of collaborative preference scores as the approximate probability distribution.

[0243] In some embodiments, the distribution of collaborative preference scores is shown as follows.

[0244]

[0245] In the formula, |I| represents the total number of training samples, represents the preference score of sample object u for all training samples in terms of feedback behavior.

[0246] In some embodiments, the distribution of multi-modal preference scores is shown as follows.

[0247]

[0248] In the formula, |I| represents the total number of training samples, represents the preference score of sample object u for all training samples in terms of multi-modal content.

[0249] In some embodiments, the first KL divergence is The second KL divergence is

[0250] Step 650: Calculate the sum of the first KL divergence and the second KL divergence as the second loss.

[0251] In some embodiments, the loss function corresponding to the second loss is shown as follows.

[0252]

[0253] In the formula, represents the distribution of multi-modal preference scores of user u; represents the distribution of collaborative preference scores of user u.

[0254] Exemplarily, the distributions of preference scores before and after aligning user preferences are as Figure 7As shown in the figure, after learning the distributions corresponding to the collaborative preference scores and the multimodal preference scores through KL divergence, the difference between the distributions corresponding to the collaborative preference scores and the multimodal preference scores is reduced.

[0255] In summary, the method provided by the embodiment of the present application uses KL divergence to minimize the difference between the probability distributions corresponding to the collaborative preference scores and the multimodal preference scores, so that the preference scores calculated based on the collaborative feature vectors and the multimodal feature vectors can tend to be consistent, thereby improving the recognition accuracy of the multimodal model.

[0256] ·Calculation of the third loss

[0257] Step 710: Classify at least two training samples to obtain a multimodal similarity set, a unimodal similarity set, and a dissimilarity set.

[0258] In some embodiments, based on the similarity information of at least two training samples, the at least two training samples are classified to obtain a multimodal similarity set, a unimodal similarity set, and a set of dissimilar items.

[0259] In some embodiments, the similarity information is calculated based on the multimodal feature vectors of every two training samples. The calculation methods of the similarity information include at least one of cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, and Jaccard similarity.

[0260] In some embodiments, based on the similarity information of at least two training samples, using the KNN method, the K training samples with the highest multimodal similarity are obtained to obtain a multimodal similarity set In the remaining training samples, the K training samples with the highest unimodal similarity are obtained again using KNN to obtain a unimodal similarity set T i m Finally, the remaining training samples form a dissimilarity set

[0261] Step 720: Use the training samples in the multimodal similarity set as positive samples and the training samples in the unimodal similarity set and the dissimilarity set as negative samples for the first contrastive learning.

[0262] In some embodiments, using the training samples in the multimodal similarity set as positive samples and the training samples in the unimodal similarity set and the dissimilarity set as negative samples for the first contrastive learning, so that the training samples in the multimodal similarity set are distinguishable from the training samples in the unimodal similarity set and the dissimilarity set in the feature space.

[0263] Step 730: Calculate the loss of the first contrastive learning as the third loss.

[0264] In some embodiments, the loss function corresponding to the third loss is as follows.

[0265]

[0266] In the formula, represents the multimodal similarity set; represents the unimodal similarity set; represents the dissimilarity set; represents the feature vector of training sample i on modality m; represents the feature vector of training sample j on modality m.

[0267] In summary, the method provided by the embodiments of the present application can enable the multimodal model to learn to distinguish the differences between the multimodal similarity set, the unimodal similarity set, and the dissimilarity set as much as possible through contrastive learning, so as to achieve better results when the multimodal model performs feature extraction and similarity calculation.

[0268] · Calculation of the fourth loss

[0269] Step 810: Classify at least two training samples to obtain a unimodal similarity set and a set of dissimilar items.

[0270] In some embodiments, based on the similarity information of at least two training samples, classify at least two training samples to obtain a unimodal similarity set and a set of dissimilar items.

[0271] In some embodiments, the similarity information is calculated based on the multimodal feature vectors of every two training samples. The calculation methods of the similarity information include at least one of cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, and Jaccard similarity.

[0272] In some embodiments, based on the similarity information of at least two training samples, use the KNN method to obtain the K training samples with the highest unimodal similarity, and obtain the unimodal similarity set The finally remaining training samples form the dissimilarity set

[0273] Step 820: Use the training samples in the unimodal similarity set as positive samples and the training samples in the set of dissimilar items as negative samples to perform the second contrastive learning.

[0274] In some embodiments, use the training samples in the unimodal similarity set as positive samples and the training samples in the dissimilarity set as negative samples to perform the second contrastive learning, so that the training samples in the unimodal similarity set are distinguishable from the training samples in the dissimilarity set in the feature space.

[0275] Step 830: Calculate the loss of the second contrastive learning as the fourth loss.

[0276] In some embodiments, the loss function corresponding to the fourth loss is as follows.

[0277]

[0278] In the formula, represents the unimodal similarity set; represents the dissimilar set; represents the feature vector of training sample i on modality m; represents the feature vector of training sample j on modality m.

[0279] In summary, the method provided by the embodiments of the present application can enable the multimodal model to learn to distinguish the difference between the unimodal similarity set and the dissimilar set as much as possible through contrastive learning, so as to achieve better results when the multimodal model performs feature extraction and similarity calculation.

[0280] Exemplarily, the methods corresponding to the third loss and the fourth loss can be executed simultaneously. As Figure 8 shown, through the first contrastive learning and the second contrastive learning, the multimodal model can distinguish three situations: multimodal similarity, unimodal similarity, and dissimilarity.

[0281] In some embodiments, steps 410 and 430 in the model training method of the multimodal model are similar to steps 210 to 230 in the multimodal recommendation method, and will not be elaborated here. It should be noted that the method of denoising the multimodal feature vectors of at least two items to be recommended in the multimodal recommendation method can be used in the model training method of the multimodal model, that is, the model training method of the multimodal model can also implement the method corresponding to step 221 above, and / or the methods corresponding to steps 310 and 320, and / or the method corresponding to step 311, and / or the method corresponding to step 312, and / or the methods corresponding to steps 321 and 322, and / or the methods corresponding to steps 330 and 340, and / or the method corresponding to step 331, and / or the methods corresponding to steps 341 and 342. When executing at least one of the above methods, the item to be recommended in the multimodal recommendation method is equivalent to the training sample in the model training method of the multimodal model, and the object to be recommended in the multimodal recommendation method is equivalent to the sample object in the model training method of the multimodal model.

[0282] Figure 9 shows a schematic diagram of the model training method of the multimodal model provided by an exemplary embodiment of the present application. The training method of the multimodal model includes the following several modules. Next, an example will be given with the sample object being a user and the training sample being an item.

[0283] 1. Construction of Denoised Item-Item Graph

[0284] The Item-Item Graph is an important data structure used to represent the similarity or correlation degree between items. The Item-Item Graph is usually constructed from a similarity matrix or a correlation matrix, where each node represents an item and each edge represents the similarity or correlation degree between two items.

[0285] 1.1 Item-Item Semantic Graph (Similarity)

[0286] Item-Item Semantic Graph: In the Item-Item Graph, the similarity matrix is usually used to represent the similarity between items. This similarity can be content-based similarity (such as similarity in text, image, and other modalities) or object behavior-based similarity (such as user ratings and click counts for different items).

[0287] First, initialize the similarity matrix S m , where each element represents the similarity between two items in modality m.

[0288] In some embodiments, the similarity measurement methods between two items include at least one of cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, and Jaccard similarity. In the embodiments of the present application, cosine similarity is used as an example for illustration, but the similarity measurement method is not limited.

[0289] The calculation formula for the similarity between two items is as follows.

[0290]

[0291] In the formula, represents the modality feature vector of item i in modality m.

[0292] Based on the average similarity corresponding to modality m, prune the Item-Item Semantic Graph.

[0293] The calculation formula for the average similarity corresponding to modality m is as follows.

[0294]

[0295] In the formula, |I| represents the total number of items; represents the similarity between item i and item j in modality m.

[0296] The way to prune the similarity matrix is: if then set

[0297] In addition, based on the similarity information between two items in another modality m ′ the similarity between the two items in modality m is updated. That is, if

[0298] then then

[0299] For the similarity matrix, update the K items most similar to each item. The resulting adjacency matrix corresponding to the item-item semantic graph is as follows.

[0300]

[0301] In the formula, represents the list of the top K items with the highest similarity to item i; represents the similarity between item i and item j in modality m.

[0302] 1.2 Item-Item Behavior Graph (Association Degree)

[0303] The association degree matrix is usually used to represent the degree of association between items. This association degree can be a content-based association degree (such as the association degree of modalities like text, image, etc.), or an object behavior-based association degree (such as behaviors like a user's simultaneous purchase of different items, co-browsing, etc.).

[0304] First, initialize an association degree matrix S c , where each element records the frequency of the same user clicking on two items.

[0305] Based on the association degree between two items, denoise the item-item behavior graph. The noise in the item-item behavior graph usually comes from the user's accidental touch. Therefore, infrequent elements can be pruned to achieve the purpose of denoising. For example, prune the elements of . The resulting adjacency matrix corresponding to the item-item behavior graph is as follows.

[0306]

[0307] In the formula, represents the association degree between item i and item j; represents the list of the top K items with the highest association degree to item i.

[0308] 1.3 Extraction of Object Feature Vectors and Item Feature Vectors

[0309] First, perform graph convolution on the above-constructed item-item semantic graph and item-item behavior graph to obtain corresponding multi-modal feature vectors, represents the visual feature vector of item i, The text feature vector representing item i, The behavioral feature vector representing item i.

[0310] Meanwhile, based on the feedback behavior of the user, using a collaborative filtering model, the object feature vector u and the collaborative feature vector

[0311] The final item feature vector is

[0312] In the formula, Meanpooling is the average pooling method; The visual feature vector representing item i, The text feature vector representing item i, The behavioral feature vector representing item i, The collaborative feature vector.

[0313] In some embodiments, the multi-modal feature vector can be referred to as the multi-modal representation, and can also be referred to as the item representation; the visual feature vector can be referred to as the visual representation; the text feature vector can be referred to as the text representation; the behavioral feature vector can be referred to as the behavioral representation; the object feature vector can be referred to as the user representation; the collaborative feature vector can be referred to as the collaborative representation, and can also be referred to as the item ID representation.

[0314] 2. User feedback denoising

[0315] The feedback behavior of the user can include various forms, such as clicking, purchasing, commenting, rating, etc. These feedback behaviors can express the direct reaction of the user to the recommended results.

[0316] The noise in the feedback behavior of the user is manifested as the user being attracted by a certain modality of the item, but finally finding that the degree of preference for the item is not high.

[0317] The feedback behavior of the user is usually used in the recommendation link of the multi-modal recommendation system, that is, based on the feedback behavior of the user, a recommendation list is generated, and this recommendation list is used to recommend items to the user. The recommendation list is usually obtained by sorting the preferences of the user for the items. Optionally, the sorting method adopts the Bayesian personalized method.

[0318] When performing Bayesian personalized recommendation, there is an observed triple <u, i, j>. The observed triple <u, i, j> indicates that the preference of user u for item i is greater than item j, and this observed triple is the training object of Bayesian personalized recommendation.

[0319] And the probability of this observed triple <u, i, j> follows a parameterized Bernoulli distribution f(u, i), where f(u, i) represents that in the true user feedback behavior is y ui >y ujIn the case where the multi-modal model predicts that the probability of user u's preference for item i is greater than that for item j, it can be understood as the probability that the observed triple is predicted as the true preference, as shown in the following formula.

[0320]

[0321] In the formula, μ ui represents the expectation of the average preference score of user u for item i in each modality, and μ ui ∈(); represents the variance of the preference score of user u for item i in each modality; α and β are hyperparameters, which are parameters predefined before starting learning in machine learning.

[0322] Among them, the calculation formula of the expectation μ ui is shown in the following formula.

[0323]

[0324] In the formula, m represents modality m; M represents all modalities of item i; represents the preference score of user u for item i in modality m; |M| represents the total number of modalities of item i.

[0325] The variance The calculation formula is shown in the following formula.

[0326]

[0327] In the formula, m represents modality M; M represents all modalities of training sample i; represents the preference score of sample object u for training sample i in modality m; |M| represents the number of modalities; μ ui represents the expectation of the average preference score of sample object u for training sample i in each modality.

[0328] The predicted preference score of user u for item i in modality m The calculation formula is shown in the following formula.

[0329]

[0330] In the formula, u u represents the object feature vector of user u; (u u ) T represents the transpose of the object feature vector of user u; represents the item feature vector of item i in modality m.

[0331] From the calculation formula of the above f(u, i), it can be obtained that the greater the expectation μ ui , the greater the variance The smaller it is, the larger the obtained f(u, i) is. That is, the greater the average preference score of user u for item i in each modality, the more consistent the preference scores between different modalities, the more reliable the observed triple, which also means the stronger the degree to which the user is attracted by the multi-modal content of the item.

[0332] The probability distribution of the observed triples is as follows.

[0333]

[0334] In the formula, <u, i, j> ∈ D represents the observed triple. y ui represents the preference score of user u for item i, and can also be understood as the true preference score of user u for item i; y uj represents the preference score of user u for item j, and can also be understood as the true preference score of user u for item j; θ represents the model parameters of the multi-modal model.

[0335] Regarding the noise in the user's feedback behavior, the corresponding observed triple follows a parameterized Bernoulli distribution g(u, i), and g(u, i) represents the probability that in the true user feedback behavior is y ui <y uj the multi-modal model predicts that the preference of user u for item i is greater than the preference for item j, and can be understood as the probability that the observed triple is predicted as a noise preference. The probability distribution of the observed triples is as follows.

[0336]

[0337] In the formula, <u, i, j> ∈ D represents the observed triple. y ui represents the preference score of user u for item i, and can also be understood as the true preference score of user u for item i; y uj represents the preference score of user u for item j, and can also be understood as the true preference score of user u for item j; θ represents the model parameters of the multi-modal model.

[0338] The noise in the user's feedback behavior represents that the noise in the user's feedback behavior shows that the user is attracted by a certain modality of the item, but finally finds that the preference degree for the item is not high.

[0339]

[0340] In the formula, represents the maximum predicted preference score of user u for item i in any modality; represents the average preference score of the user for other items except item i; γ is a hyperparameter; μ ui represents the expectation of the average preference score of sample object u for training sample i in each modality.

[0341] Among them, is calculated as follows.

[0342]

[0343] In the formula, B represents the current training batch; represents the predicted preference score of user u for item j; |B| represents the number of training samples in the current training batch.

[0344] Thus, the loss function corresponding to the Bayesian personalized ranking method can be obtained.

[0345]

[0346] In the formula, f(u,i) represents the probability that the observed triple is predicted as the true preference; represents the predicted preference score of sample object u for training sample i; represents the predicted preference score of sample object u for training sample j; g(u,i) represents the probability that the observed triple is predicted as the noisy preference; θ represents the model parameters of the multimodal model.

[0347] 3. User preference alignment

[0348] Alignment is an important step in the multimodal system. The purpose of alignment is to determine the connection and interaction between different modal data, so as to better utilize the complementarity of multimodal data. Through alignment, the association between different modal data can be established.

[0349] User preference is an important data in the multimodal recommendation system, and since user preference is relatively stable data, it can be used for alignment.

[0350] The user preference distribution includes the preference distribution on multimodal content and the preference distribution on feedback behavior.

[0351] Preference distribution on multimodal content is as follows.

[0352]

[0353] In the formula, |I| represents the total number of items, represents the predicted preference score of user u for all items on multimodal content.

[0354] Predicted preference score on multimodal content The calculation formula is as follows.

[0355]

[0356] In the formula, uu The object feature vector for user u; The item feature vector of item i on the multimodal content.

[0357] In some embodiments, the item feature vector on the multimodal content includes a visual feature vector A text feature vector A behavior feature vector The item feature vectors on each modality can be aggregated by average pooling.

[0358]

[0359] The preference distribution in the feedback behavior Is as follows.

[0360]

[0361] In the formula, |I| represents the total number of items, Represents the predicted preference score of user u for all items in the feedback behavior.

[0362] The predicted preference score in the feedback behavior The calculation formula is as follows.

[0363]

[0364] In the formula, u u Is the object feature vector of user u; Is the feature vector of item i in the feedback behavior, and can also be called the collaborative feature vector.

[0365] In some embodiments, the item feature vector in the feedback behavior can be extracted using a collaborative filtering method.

[0366] Finally, the KL divergence is used to align the two distributions.

[0367]

[0368] In the formula, Represents the distribution of the multimodal preference scores of user u; Represents the distribution of the collaborative preference scores of user u.

[0369] The KL divergence is a method for measuring the difference between two probability distributions, and alignment can be achieved by minimizing the KL divergence, that is, minimizing the difference between the two probability distributions to achieve alignment.

[0370] 4. Alignment of item hierarchical relationships

[0371] In the related art, the features of different modalities of an item are usually aligned so that the feature vectors of different modalities are in the same feature space. However, when aligning items, alignment can also be considered from the aspect of the similarity of subdivided items. For example, items can be subdivided into multimodal similarity, unimodal similarity, and dissimilarity.

[0372] First, similarity calculation is performed. The similarity calculation formulas for each modality are as shown in the above 1.1 and will not be elaborated here.

[0373] The multimodal similarity calculation formula is as follows.

[0374]

[0375] In the formula, M is a set of at least one modality other than modality m; represents the similarity between item i and item j in modality m. It should be noted that for item i and item j, there are at least one multimodal similarity, and the multimodal similarity is used to indicate the multimodal similarity between item i and item j in modality m. In different modalities, the multimodal similarity between item i and item j may be different.

[0376] After calculating the unimodal similarity and multimodal similarity, set partitioning is performed. Using the KNN method, the K items with the highest multimodal similarity are obtained to form the multimodal similar item set In the remaining item set, the KNN method is used again to obtain the K items with the highest unimodal similarity to form the unimodal similar set The finally remaining items form the dissimilar item set

[0377] Then, the contrastive learning method is used to align the subdivided sets.

[0378] The loss function when aligning the multimodal similar set and the unimodal similar set.

[0379]

[0380] In the formula, represents the multimodal similar set; represents the unimodal similar set; represents the dissimilar set; represents the feature vector of training sample i in modality m; represents the feature vector of training sample j in modality m.

[0381] The loss function when aligning the multimodal similar set and the dissimilar item set.

[0382]

[0383] In the formula, represents a unimodal similarity set; represents a dissimilar set; represents the feature vector of training sample i on modality m; represents the feature vector of training sample j on modality m.

[0384] The loss function of the alignment method for obtaining the subdivided set is as follows.

[0385] L AI = L 1I-MM + L AI-s

[0386] During the training of the multimodal model, the final loss function is:

[0387] L = L D-BPR + λ1L Au + λ2L AI

[0388] In the formula, λ1 and λ2 are two hyperparameters.

[0389] In some embodiments, the above multimodal model can be applied to recommendation systems such as look-alike recommendation, official account recommendation, video account recommendation, and product recommendation.

[0390] Please refer to Figure 10 , which shows the structural block diagram of the multimodal recommendation device provided by an exemplary embodiment of the present application. This device has the functions of implementing the above multimodal recommendation method example, and the functions can be implemented by hardware or by hardware executing corresponding software. This device can be the server introduced above or can be set in the server. As Figure 10 shown, this device may include: a first acquisition module 910, a first denoising module 920, a first calculation module 930, and a first generation module 940.

[0391] The first acquisition module 910 is used to acquire the multimodal feature vectors of at least two items to be recommended, and the multimodal feature vector of each item to be recommended includes the feature vectors of each item to be recommended on at least two modalities; and to acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended.

[0392] The first denoising module 920 is used to denoise the multimodal feature vectors of the at least two items to be recommended to obtain the denoised multimodal feature vectors of the at least two items to be recommended.

[0393] The first calculation module 930 is configured to calculate a preference score of the object to be recommended for each of the items to be recommended based on the denoised multi-modal feature vector and the object feature vector.

[0394] The first generation module 940 is configured to generate the recommendation list of the object to be recommended based on the preference scores of each of the items to be recommended, where the recommendation list includes at least one of the items to be recommended.

[0395] In some embodiments, the first denoising module 920 includes a first denoising sub-module.

[0396] The first denoising sub-module is configured to denoise the multi-modal feature vectors of the at least two items to be recommended based on at least one of the similarity information and the correlation information of the at least two items to be recommended, so as to obtain the denoised multi-modal feature vectors of the at least two items to be recommended;

[0397] wherein, the similarity information is used to indicate the similarity degree of two items to be recommended in the same modality, and the correlation information is used to indicate the correlation degree between two items to be recommended.

[0398] In some embodiments, the first denoising sub-module includes a first denoising unit and a first aggregation unit.

[0399] The first denoising unit is configured to denoise the similarity information between every two of the at least two items to be recommended, so as to obtain the denoised similarity information;

[0400] The first aggregation unit is configured to aggregate the multi-modal feature vector of a first item to be recommended and the multi-modal feature vector of a second item to be recommended based on the denoised similarity information, so as to obtain the denoised multi-modal feature vector of the first item to be recommended;

[0401] wherein, the first item to be recommended is any one of the at least two items to be recommended, and the second item to be recommended is an item to be recommended whose similarity information with the first item to be recommended is not zero.

[0402] In an alternative embodiment, the first denoising unit is further configured to set the first similarity to zero when the first similarity is less than the average similarity of the at least two items to be recommended in the first modality; wherein, the first similarity is used to indicate the similarity of the first item to be recommended and a third item to be recommended in the first modality.

[0403] In an optional embodiment, the first denoising unit is further configured to set the first similarity to zero when the second similarity is zero; wherein, the first similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the first modality, the second similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the second modality, and the second modality is the modality other than the first modality among the at least two modalities.

[0404] In an optional embodiment, the first aggregation unit is further configured to aggregate the feature vectors of the first item to be recommended and the second item to be recommended in the i-th modality with the similarity between the first item to be recommended and the second item to be recommended in the i-th modality as the weight, to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer.

[0405] In an optional embodiment, the first aggregation unit is further configured to repeat the above step for other modalities different from the i-th modality, to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0406] In some embodiments, the first denoising sub-module further includes a second denoising unit and a second aggregation unit.

[0407] The second denoising unit is configured to denoise the correlation information between every two of the at least two items to be recommended, to obtain the denoised correlation information;

[0408] The second aggregation unit is configured to aggregate the multi-modal feature vector of the first item to be recommended and the multi-modal feature vector of the fourth item to be recommended based on the denoised correlation information, to obtain the denoised multi-modal feature vector of the first item to be recommended;

[0409] wherein, the fourth item to be recommended is an item to be recommended whose correlation information with the first item to be recommended is not zero.

[0410] In an optional embodiment, the second denoising unit is further configured to set the correlation between the first item to be recommended and the fifth item to be recommended to zero when the correlation between the first item to be recommended and the fifth item to be recommended is lower than the correlation threshold.

[0411] In an optional embodiment, the second aggregation unit is further configured to aggregate the feature vectors of the first item to be recommended and the fourth item to be recommended in the i-th modality with the similarity between the first item to be recommended and the fourth item to be recommended in the i-th modality as the weight, to obtain the denoised feature vector of the first item to be recommended in the i-th modality; i is a positive integer.

[0412] In an optional embodiment, the second aggregation unit is further configured to repeat the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first item to be recommended.

[0413] In some embodiments, the first acquisition module further includes a first acquisition sub-module.

[0414] The first acquisition sub-module is configured to extract the multi-modal feature vectors of the at least two items to be recommended based on a multi-modal model.

[0415] Please refer to Figure 11 , which shows a structural block diagram of a model training device for a multi-modal model provided by an exemplary embodiment of the present application. The device has the function of implementing the above example of the training method of the multi-modal model. The function can be implemented by hardware or by hardware executing corresponding software. The device can be the server introduced above or can be set in the server. As Figure 11 shown, the device may include: a second acquisition module 1010, a second denoising module 1020, a second calculation module 1030, and a first training module 1040.

[0416] The second acquisition module 1010 is configured to acquire the multi-modal feature vectors of at least two training samples, where the multi-modal feature vector of each training sample includes the feature vectors of the training sample in at least two modalities; and acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object.

[0417] The second denoising module 1020 is configured to denoise the multi-modal feature vectors of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples.

[0418] The second calculation module 1030 is configured to calculate the predicted preference scores of the sample object for each of the training samples based on the denoised multi-modal feature vectors and the object feature vector.

[0419] The first training module 1040 is configured to train the multi-modal model based on the sample loss of the training samples.

[0420] In some embodiments, the second denoising module 1020 includes a second denoising sub-module.

[0421] The second denoising sub-module is configured to denoise the multi-modal feature vectors of the at least two training samples based on at least one of the similarity information and the correlation information of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples;

[0422] Among them, the similarity information is used to indicate the similarity degree of two training samples in the same modality, and the correlation information is used to indicate the correlation degree between two training samples.

[0423] In some embodiments, the second denoising sub-module includes a third denoising unit and a third aggregation unit.

[0424] The third denoising unit is configured to denoise the similarity information between every two of the at least two training samples to obtain denoised similarity information.

[0425] The third aggregation unit is configured to aggregate the multi-modal feature vectors of the first training sample and the second training sample based on the denoised similarity information to obtain the denoised multi-modal feature vector of the first training sample.

[0426] Among them, the first training sample is any one of the at least two training samples, and the second training sample is a training sample whose similarity information with the first training sample is not zero.

[0427] In an alternative embodiment, the third denoising unit is further configured to set the first similarity to zero when the first similarity is less than the average similarity of the at least two training samples in the first modality; wherein, the first similarity is used to indicate the similarity between the first training sample and the third training sample in the first modality.

[0428] In an alternative embodiment, the third denoising unit is further configured to set the first similarity to zero when the second similarity is zero; wherein, the first similarity is used to indicate the similarity between the first training sample and the third training sample in the first modality, the second similarity is used to indicate the similarity between the first training sample and the third training sample in the second modality, and the second modality is a modality other than the first modality among the at least two modalities.

[0429] In an alternative embodiment, the third aggregation unit is further configured to use the similarity between the first training sample and the second training sample in the i-th modality as a weight to aggregate the feature vectors of the first training sample and the second training sample in the i-th modality to obtain the denoised feature vector of the first training sample in the i-th modality; i is a positive integer.

[0430] In an alternative embodiment, the third aggregation unit is further configured to repeat the above step for other modalities different from the i-th modality to obtain the denoised multi-modal feature vector of the first training sample.

[0431] In some embodiments, the second denoising sub-module further includes a fourth denoising unit and a fourth aggregation unit.

[0432] A fourth denoising unit, configured to denoise the correlation information between every two of the at least two training samples to obtain denoised correlation information.

[0433] A fourth aggregation unit, configured to aggregate the multimodal feature vector of the first training sample and the multimodal feature vector of the fourth training sample based on the denoised correlation information to obtain the denoised multimodal feature vector of the first training sample;

[0434] wherein the fourth training sample is a training sample whose correlation information with the first training sample is not zero.

[0435] In an optional embodiment, the fourth denoising unit is further configured to set the correlation between the first training sample and the fifth training sample to zero when the correlation between the first training sample and the fifth training sample is lower than a correlation threshold.

[0436] In an optional embodiment, the fourth aggregation unit is further configured to use the similarity between the first training sample and the fourth training sample in the i-th modality as a weight to aggregate the feature vectors of the first training sample and the fourth training sample in the i-th modality to obtain the denoised feature vector of the first training sample in the i-th modality; i is a positive integer.

[0437] In an optional embodiment, the fourth aggregation unit is further configured to repeat the above step for other modalities different from the i-th modality to obtain the denoised multimodal feature vector of the first training sample.

[0438] In some embodiments, the sample loss includes the first loss, and the apparatus further includes a first classification module, a third acquisition module, a third calculation module, and a fourth calculation module.

[0439] The first classification module is configured to classify the at least two training samples based on the feedback behaviors of the sample object on the at least one training sample to obtain positive samples and negative samples, where the positive samples are training samples with feedback behaviors, and the negative samples are training samples without feedback behaviors.

[0440] The third acquisition module is configured to acquire an observation triple for training, where the observation triple includes the sample object, the positive sample, and the negative sample, and the observation triple is used to indicate that the sample object has a greater preference for the positive sample than for the negative sample.

[0441] A third computing module, configured to calculate the probability that the observed triple is predicted as a true preference and the probability that the observed triple is predicted as a noise preference, where the true preference is used to indicate that the predicted preference score of the positive sample is greater than the predicted preference score of the negative sample, and the noise preference is used to indicate that the predicted preference score of the positive sample is less than the predicted preference score of the negative sample.

[0442] A fourth computing module, configured to calculate the first loss, where the first loss is the sum of a first product value and a second product value. The first product value is the product of the probability that the observed triple is predicted as the true preference and a first difference value, and the second product value is the product of the probability that the observed triple is predicted as the noise preference and a second difference value. The first difference value is the difference between the predicted preference score of the positive sample and the predicted preference score of the negative sample, and the second difference value is the difference between the predicted preference score of the negative sample and the predicted preference score of the positive sample.

[0443] In some embodiments, the sample loss includes a second loss, and the apparatus further includes a fourth acquisition module, a fifth computing module, a sixth computing module, a seventh computing module, and an eighth computing module.

[0444] A fourth acquisition module, configured to acquire the collaborative feature vectors of the at least two training samples, where the collaborative feature vectors are used to indicate the feature vectors of the sample object for the feedback behaviors of the at least two training samples.

[0445] A fifth computing module, configured to calculate the collaborative preference score of the sample object for each training sample based on the collaborative feature vectors of the at least two training samples and the object feature vector of the sample object.

[0446] A sixth computing module, configured to calculate the multimodal preference score of the sample object for each training sample based on the multimodal feature vectors of the at least two training samples and the object feature vector of the sample object.

[0447] A seventh computing module, configured to calculate a first KL divergence and a second KL divergence. The first KL divergence is the KL divergence calculated with the distribution of the collaborative preference scores as the original probability distribution and the distribution of the multimodal preference scores as the approximate probability distribution; the second KL divergence is the KL divergence calculated with the distribution of the multimodal preference scores as the original probability distribution and the distribution of the collaborative preference scores as the approximate probability distribution.

[0448] An eighth computing module, configured to calculate the sum of the first KL divergence and the second KL divergence as the second loss.

[0449] In some embodiments, the sample loss includes the third loss, and the apparatus further includes a second classification module, a first learning module, and a ninth calculation module.

[0450] The second classification module is configured to classify the at least two training samples to obtain a multimodal similarity set, a unimodal similarity set, and a dissimilarity set.

[0451] The first learning module is configured to perform first contrastive learning using the training samples in the multimodal similarity set as positive samples and the training samples in the unimodal similarity set and the dissimilarity set as negative samples.

[0452] The ninth calculation module is configured to calculate the loss of the first contrastive learning as the third loss.

[0453] In some embodiments, the sample loss includes the fourth loss, and the apparatus further includes a third classification module, a second learning module, and a tenth calculation module.

[0454] The third classification module is configured to classify the at least two training samples to obtain a unimodal similarity set and a dissimilarity set.

[0455] The second learning module is configured to perform second contrastive learning using the training samples in the unimodal similarity set as positive samples and the training samples in the dissimilarity set as negative samples.

[0456] The tenth calculation module is configured to calculate the loss of the second contrastive learning as the fourth loss.

[0457] It should be noted that when the apparatus provided in the above embodiments implements its functions, only the above-mentioned division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the method embodiments belong to the same concept, and the specific implementation process thereof can be seen in the method embodiments, which will not be elaborated here.

[0458] Figure 12 The block diagram of a computer device provided by an exemplary embodiment of the present application is shown.

[0459] The computer device 1100 includes a Central Processing Unit (CPU) 1101, a system memory 1104 including a Random Access Memory (RAM) 1102 and a Read-Only Memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 further includes a Basic Input / Output System (Input / Output system, I / O system) 1106 for facilitating information transfer between various components within the computer device, and a mass storage device 1107 for storing an operating system 1113, application programs 1114, and other program modules 1115.

[0460] The Basic Input / Output System 1106 includes a display 1108 for displaying information and input devices 1109 such as a mouse, keyboard, etc. for user input of information. The display 1108 and the input devices 1109 are both connected to the central processing unit 1101 through an input / output controller 1110 connected to the system bus 1105. The Basic Input / Output System 1106 may further include an input / output controller 1110 for receiving and processing inputs from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0461] The mass storage device 1107 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable storage medium provide non-volatile storage for the computer device 1100. That is to say, the mass storage device 1107 may include a computer-readable storage medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0462] Without loss of generality, the computer-readable storage medium may include a computer storage medium and a communication medium. The computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. The computer storage medium includes RAM, ROM, erasable programmable read-only registers (EPROM), electrically-erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cartridges, tapes, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above several types. The above-mentioned system memory 1104 and mass storage device 1107 can be collectively referred to as memory.

[0463] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1101. The one or more programs include instructions for implementing the above method embodiments. The central processing unit 1101 executes the one or more programs to implement the methods provided by the above various method embodiments.

[0464] According to various embodiments of the present application, the computer device 1100 can also run by connecting to a remote computer device on the network through a network such as the Internet. That is, the computer device 1100 can be connected to the network 1112 through the network interface unit 1111 connected to the system bus 1105. Or rather, the network interface unit 1111 can also be used to connect to other types of networks or remote computer device systems (not shown).

[0465] The memory further includes one or more programs, and the one or more programs are stored in the memory. The one or more programs include steps for performing the method executed by the terminal device in the method provided by the embodiments of the present application.

[0466] In an exemplary embodiment, a non-transitory computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above multi-modal recommendation method and / or the model training method of the multi-modal model.

[0467] In an exemplary embodiment, a computer program product is further provided, which is used to implement the above multi-modal recommendation method and / or the model training method of the multi-modal model when executed by a processor.

[0468] It should be understood that "a plurality of" as mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. In addition, the step numbers described herein only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers may be executed simultaneously, or two steps with different numbers may be executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0469] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A multimodal recommendation method, characterized in that, The method includes: Obtaining multi-modal feature vectors of at least two items to be recommended, where the multi-modal feature vector of each item to be recommended includes feature vectors of each item to be recommended in at least two modalities; and obtaining an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended; Denosing the multi-modal feature vectors of the at least two items to be recommended to obtain denoised multi-modal feature vectors of the at least two items to be recommended; Calculating the preference scores of the object to be recommended for each of the items to be recommended based on the denoised multi-modal feature vectors and the object feature vector; Generating the recommendation list of the object to be recommended based on the preference scores of each of the items to be recommended, where the recommendation list includes at least one of the items to be recommended.

2. The method according to claim 1, characterized in that, The denosing the multi-modal feature vectors of the at least two items to be recommended to obtain denoised multi-modal feature vectors of the at least two items to be recommended includes: Denosing the multi-modal feature vectors of the at least two items to be recommended based on at least one of the similarity information and the correlation information of the at least two items to be recommended to obtain denoised multi-modal feature vectors of the at least two items to be recommended; Wherein, the similarity information is used to indicate the similarity degree of two items to be recommended in the same modality, and the correlation information is used to indicate the correlation degree between two items to be recommended.

3. The method according to claim 2, characterized in that, The denosing the multi-modal feature vectors of the at least two items to be recommended based on at least one of the similarity information and the correlation information of the at least two items to be recommended to obtain denoised multi-modal feature vectors of the at least two items to be recommended includes: Denosing the similarity information between every two of the at least two items to be recommended to obtain denoised similarity information; Aggregating the multi-modal feature vector of the first item to be recommended and the multi-modal feature vector of the second item to be recommended based on the denoised similarity information to obtain the denoised multi-modal feature vector of the first item to be recommended; Wherein, the first item to be recommended is any one of the at least two items to be recommended, and the second item to be recommended is an item to be recommended whose similarity information with the first item to be recommended is not zero.

4. The method according to claim 3, characterized in that, The denosing the similarity information between every two of the at least two items to be recommended to obtain denoised similarity information includes: Setting the first similarity to zero when the first similarity is less than the average similarity of the at least two items to be recommended in the first modality; Wherein, the first similarity is used to indicate the similarity of the first item to be recommended and the third item to be recommended in the first modality.

5. The method according to claim 3, characterized in that, The denosing the similarity information between every two of the at least two items to be recommended to obtain denoised similarity information includes: Setting the first similarity to zero when the second similarity is zero; Wherein, the first similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the first modality, and the second similarity is used to indicate the similarity between the first item to be recommended and the third item to be recommended in the second modality, where the second modality is the modality other than the first modality among the at least two modalities.

6. The method according to claim 2, characterized in that, Denosing the multi-modal feature vectors of the at least two items to be recommended based on at least one of the similarity information and the correlation information of the at least two items to be recommended to obtain denoised multi-modal feature vectors, including: Denosing the correlation information between each two of the at least two items to be recommended to obtain denoised correlation information; Aggregating the multi-modal feature vectors of the first item to be recommended and the fourth item to be recommended based on the denoised correlation information to obtain the denoised multi-modal feature vector of the first item to be recommended; Wherein, the fourth item to be recommended is an item to be recommended whose correlation information with the first item to be recommended is not zero.

7. The method according to claim 6, characterized in that, The denosing the correlation information between each two of the at least two items to be recommended to obtain denoised correlation information includes: When the correlation between the first item to be recommended and the fifth item to be recommended is lower than the correlation threshold, setting the correlation between the first item to be recommended and the fifth item to be recommended to zero.

8. A model training method for a multimodal model, characterized in that, The method includes: Obtaining the multi-modal feature vectors of at least two training samples, where the multi-modal feature vector of each training sample includes the feature vectors of the training sample in at least two modalities; and obtaining an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object; Denosing the multi-modal feature vectors of the at least two training samples to obtain the denoised multi-modal feature vectors of the at least two training samples; Calculating the predicted preference scores of the sample object for each of the training samples based on the denoised multi-modal feature vectors and the object feature vector; Training the multi-modal model based on the sample loss of the training samples.

9. The method according to claim 8, characterized in that, The sample loss includes at least one of the following: The first loss, which is used to indicate the loss between the ranking of the predicted preference scores of the sample object for each of the training samples and the true preference ranking of the sample object for each of the training samples, where the true preference ranking is used to indicate the preference degree of the sample object for each of the training samples; The second loss, which is used to indicate the loss between the multi-modal preference score calculated based on the multi-modal feature vector of the training sample and the collaborative preference score calculated based on the collaborative feature vector of the training sample; The third loss, which is used to indicate the loss of contrast learning with the multi-modal similarity set as the positive sample and the single-modal similarity set and the dissimilarity set as the negative samples; The fourth loss, which is used to indicate the loss of contrast learning with the single-modal similarity set as the positive sample and the dissimilarity set as the negative sample.

10. The method according to claim 9, wherein The sample loss includes the first loss, and the method further includes: Classify the at least two training samples based on the feedback behavior of the sample object on at least one of the training samples to obtain positive samples and negative samples, where the positive samples are the training samples with feedback behavior, and the negative samples are the training samples without feedback behavior; Obtain observation triples for training, where the observation triples include the sample object, the positive samples, and the negative samples, and the observation triples are used to indicate that the sample object has a greater preference for the positive samples than for the negative samples; Calculate the probability that the observation triples are predicted as true preferences and the probability that the observation triples are predicted as noisy preferences, where the true preference is used to indicate that the predicted preference score of the positive samples is greater than the predicted preference score of the negative samples, and the noisy preference is used to indicate that the predicted preference score of the positive samples is less than the predicted preference score of the negative samples; Calculate the first loss, where the first loss is the sum of a first product value and a second product value. The first product value is the product of the probability that the observation triples are predicted as the true preference and a first difference, and the second product value is the product of the probability that the observation triples are predicted as the noisy preference and a second difference. The first difference is the difference between the predicted preference score of the positive samples and the predicted preference score of the negative samples, and the second difference is the difference between the predicted preference score of the negative samples and the predicted preference score of the positive samples.

11. The method according to claim 9, wherein The sample loss includes the second loss, and the method further includes: Obtain the collaborative feature vectors of the at least two training samples, where the collaborative feature vectors are used to indicate the feature vectors of the sample object's feedback behavior on the at least two training samples; Calculate the collaborative preference scores of the sample object for each training sample based on the collaborative feature vectors of the at least two training samples and the object feature vectors of the sample object; Calculate the multimodal preference scores of the sample object for each training sample based on the multimodal feature vectors of the at least two training samples and the object feature vectors of the sample object; Calculate a first KL divergence and a second KL divergence. The first KL divergence is the KL divergence calculated with the distribution of the collaborative preference scores as the original probability distribution and the distribution of the multimodal preference scores as the approximate probability distribution; the second KL divergence is the KL divergence calculated with the distribution of the multimodal preference scores as the original probability distribution and the distribution of the collaborative preference scores as the approximate probability distribution; Calculate the sum of the first KL divergence and the second KL divergence as the second loss.

12. The method according to claim 9, wherein The sample loss includes the third loss, and the method further includes: Classify the at least two training samples to obtain the multimodal similarity set, the unimodal similarity set, and the dissimilarity set; Use the training samples in the multimodal similarity set as positive samples and the training samples in the unimodal similarity set and the dissimilarity set as negative samples to perform the first contrastive learning; Calculate the loss of the first contrastive learning as the third loss.

13. The method according to claim 9, wherein The sample loss includes the fourth loss, and the method further includes: Classify the at least two training samples to obtain the unimodal similarity set and the dissimilarity set; Use the training samples in the unimodal similarity set as positive samples and the training samples in the dissimilarity set as negative samples to perform second contrastive learning; Calculate the loss of the second contrastive learning as the fourth loss.

14. A multimodal recommendation device, wherein The device includes: A first acquisition module, configured to acquire multimodal feature vectors of at least two items to be recommended, where the multimodal feature vector of each item to be recommended includes feature vectors of the each item to be recommended in at least two modalities; and acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the object to be recommended; A first denoising module, configured to denoise the multimodal feature vectors of the at least two items to be recommended to obtain the denoised multimodal feature vectors of the at least two items to be recommended; A first calculation module, configured to calculate the preference score of the object to be recommended for each of the items to be recommended based on the denoised multimodal feature vectors and the object feature vector; A first generation module, configured to generate the recommendation list of the object to be recommended based on the preference score of each of the items to be recommended, where the recommendation list includes at least one of the items to be recommended.

15. A training device for a multimodal model, wherein The device includes: A second acquisition module, configured to acquire multimodal feature vectors of at least two training samples, where the multimodal feature vector of each training sample includes feature vectors of the training sample in at least two modalities; and acquire an object feature vector, where the object feature vector is used to indicate the preference feature vector of the sample object; A second denoising module, configured to denoise the multimodal feature vectors of the at least two training samples to obtain the denoised multimodal feature vectors of the at least two training samples; A second calculation module, configured to calculate the predicted preference score of the sample object for each of the training samples based on the denoised multimodal feature vectors and the object feature vector; A first training module, configured to train the multimodal model based on the sample loss of the training samples.

16. A computer device, wherein The computer device includes: a processor and a memory, where at least one program is stored in the memory; the processor is configured to execute the at least one program in the memory to implement the multimodal recommendation method as described in any one of claims 1 to 7 above, and / or, the model training method of the multimodal model as described in any one of claims 8 to 13 above.

17. A computer-readable storage medium, characterized in that, An executable instruction is stored in the computer-readable storage medium, and the executable instruction is loaded and executed by a processor to implement the multimodal recommendation method as described in any one of claims 1 to 7 above, and / or, the model training method of the multimodal model as described in any one of claims 8 to 13 above.

18. A computer program product, characterized in that, The computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement the multimodal recommendation method as described in any one of claims 1 to 7 above, and / or, the model training method of the multimodal model as described in any one of claims 8 to 13 above.

Images