A multimodal recommendation method and system based on spectral polynomial filter

Abstract

The present invention relates to the field of data processing technology, and discloses a multimodal recommendation method and system based on a spectrogram polynomial filter. The method comprises: inputting user data and project data into a PolyRec model, the PolyRec model comprising a multimodal encoder, a behavioral encoder, and a multimodal alignment module; encoding and fusing the user data and project data using the multimodal encoder to obtain a first feature; encoding the user data using the behavioral encoder to obtain a second feature; inputting the first feature and the second feature into a multimodal alignment module for fusion, and outputting a personalized recommendation result. The present invention can more comprehensively capture user preferences by integrating the user's multimodal features and behavioral information at different frequencies; using a frequency signal scaler and a unified spectrogram neural network function, the model can flexibly adapt to the characteristics of signals at different frequencies, further enhancing the robustness and adaptability of the recommendation system.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing technology, and in particular to a multimodal recommendation method and system based on a spectral polynomial filter. Background Art

[0002] In the digital age, recommender systems have become a core component of e-commerce and content delivery platforms, enhancing user experience by analyzing user behavior and providing customized content. These systems particularly benefit from the fusion of multimodal information, such as images, text, and user interaction data, which makes recommendations more precise and personalized. Early work on multimodal recommender systems enhanced traditional matrix factorization methods by integrating visual features. In recent years, graph neural networks (GNNs) have been widely used in recommender systems due to their strong ability to process graph-structured data, particularly in collaborative filtering-based recommendations. These GNN models are able to incorporate multimodal information from different views and learn complex representations of user preferences.

[0003] Although multimodal recommendation methods based on graph neural networks have achieved certain success, they still face challenges in processing user behavior information and integrating multimodal information. First, existing models often ignore high-frequency signals in user-item interaction graphs, which are crucial for capturing users' immediate preference changes. Second, these models lack effective guidance when integrating multimodal information and behavioral information of different frequencies, resulting in poor performance when processing complex user behaviors. In addition, existing models rely on specific graph signal filters and lack flexibility and uniformity, which limits their potential in capturing user preferences. Therefore, developing a recommendation system that can comprehensively consider multimodal information and user behavior information of different frequencies is of great significance to improving recommendation quality and user satisfaction. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a multimodal recommendation method based on spectral polynomial filters to solve the problems of insufficient modeling of user behavior information of different frequencies in existing multimodal recommendation systems and lack of effective guidance for the fusion of multimodal information and behavior information.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a multimodal recommendation method based on a spectral polynomial filter, comprising:

[0008] Get user data and project data;

[0009] Inputting the user data and the project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavior encoder, and a multimodal alignment module;

[0010] Encoding and fusing text modality and visual modality information in the user data and the project data using the multimodal encoder to obtain a first feature;

[0011] Encoding the user data using the behavior encoder to obtain a second feature;

[0012] The first feature and the second feature are input into the multimodal alignment module for fusion, and a personalized recommendation result is output.

[0013] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter described in the present invention, the multimodal encoder includes a pre-trained encoder, a behavior-guided purifier, an item-item view encoder and a behavior-aware fusion.

[0014] As a preferred embodiment of the multimodal recommendation method based on spectral polynomial filter of the present invention, the multimodal encoder is used to encode and fuse the text modality and visual modality information in the user and item data, and the first feature is obtained, which includes:

[0015] Using the pre-trained encoder to learn original multimodal features in the project data to obtain preliminary features;

[0016] Fusing the preliminary features with initial behavioral features in the project data, and performing denoising processing using the behavioral guidance purifier to obtain preference-related modal features;

[0017] Inputting the preference-related modal features into the item-item view encoder to perform graph convolution to obtain semantic correlation signals of multimodal features in the item;

[0018] The behavior perception fuser is used to fuse features of different modalities to obtain a first feature.

[0019] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, wherein: inputting the preference-related modal features into the item-item view encoder to perform graph convolution, and obtaining the semantic correlation signal of the multimodal features in the item includes:

[0020] Build a fully connected graph based on the raw multimodal features in the project data , use KNN algorithm to Perform sparse processing to obtain a sparse graph;

[0021] Normalizing the sparse graph to obtain an item-item similarity matrix, performing message propagation on the item-item similarity matrix using a graph neural network to obtain modal features of the items, and obtaining modal features of the users based on the modal features of the items;

[0022] The modal features of the user and the modal features of the item are connected to obtain multimodal features.

[0023] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, wherein: encoding the user data by using the behavior encoder to obtain the second feature includes:

[0024] Encoding the user data, converting the user data into a graph structure, modeling the graph structure using a unified spectral graph neural network function, capturing different frequency components in the user data, and obtaining a second feature;

[0025] A graph signal filtering function is defined to extract specific frequency components from user data. The graph signal filtering function is:

[0026] In the high frequency range is monotonically increasing, and ,in ;

[0027] In the low frequency range is monotonically increasing, and ,in ;

[0028] In the intermediate frequency range The inner is a nonlinear transformation.

[0029] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, the second feature includes:

[0030] The frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as:

[0031] ,

[0032] in, Represent the scaling functions of high-frequency, medium-frequency and low-frequency signals respectively, , .

[0033] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter described in the present invention, the PolyRec model includes:

[0034] A loss function is designed to optimize the PolyRec model. The loss function is expressed as:

[0035] ,

[0036] in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, To control the hyperparameters of the contrast auxiliary tasks separately, is a hyperparameter of the regularization effect, The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

[0037] In a second aspect, the present invention provides a multimodal recommendation system based on spectral polynomial filter, comprising:

[0038] Acquisition module, used to obtain user data and project data;

[0039] a feature acquisition module, configured to input the user data and project data into a PolyRec model, the PolyRec model comprising a multimodal encoder, a behavioral encoder, and a multimodal alignment module; encode and fuse textual modality and visual modality information in the user data and project data using the multimodal encoder to obtain a first feature; and encode the user data using the behavioral encoder to obtain a second feature;

[0040] The feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion, and output a personalized recommendation result.

[0041] In a third aspect, the present invention provides an electronic device, comprising:

[0042] memory and processor;

[0043] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal recommendation method based on the spectral polynomial filter are implemented.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal recommendation method based on spectral polynomial filter.

[0045] Compared with the existing technology, the present invention has the following beneficial effects: the multimodal recommendation method based on spectral polynomial filter proposed in the present invention, namely the PolyRec model, improves the performance of the recommendation system in processing user behavior and multimodal information; by effectively integrating the user's multimodal features and behavioral information of different frequencies, the PolyRec model can more comprehensively capture user preferences and provide more accurate personalized recommendations; the use of frequency signal scalers and unified spectral graph neural network functions enables the model to flexibly adapt to the characteristics of signals of different frequencies, further enhancing the robustness and adaptability of the recommendation system. Overall, the present invention provides an efficient recommendation solution for e-commerce platforms, which can improve user experience and platform sales efficiency while protecting user privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 Schematic diagram of the overall process of a multimodal recommendation method based on a spectral polynomial filter according to an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of the PolyRec model of the multimodal recommendation method based on spectral polynomial filter according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0050] Example 1

[0051] Reference Figure 1~Figure 2 , as one embodiment of the present invention, provides a multimodal recommendation method based on a spectral polynomial filter, comprising:

[0052] S100: Obtain user data and project data; input the user data and project data into the PolyRec model, which includes a multimodal encoder, a behavior encoder, and a multimodal alignment module; the PolyRec model is as follows Figure 2 As shown;

[0053] In this embodiment of the present application, user data includes basic user information and user behavior information, such as user ID, gender, and historical behavior records. Item data includes basic information about items clicked by users, as well as text and visual modal information about the items, such as displayed images and text descriptions of the items.

[0054] In the embodiment of the present application, firstly, an undirected graph is given ,set up and is the node set and edge set of the graph, and the user set is represented as , the item set is represented as , the node set is represented as , the interaction matrix between users and items is , where if users and Items have interactions, then . Adjacency Matrix It can be expressed as follows:

[0055] ,

[0056] in, is the interaction matrix between users and items.

[0057] It should be noted that spectral graph neural networks perform graph convolution in the spectral domain of the Laplacian matrix. The normalized Laplacian matrix can be defined as:

[0058] ,

[0059] in, represents the degree matrix of the graph, is the adjacency matrix, represents the identity matrix;

[0060] By eigenvalue decomposition, we can get its eigenvalues ​​and corresponding eigenvectors. ,in, is the eigenvector, is the corresponding eigenvalue, The eigenvalue range is ;

[0061] Signal The graphical Fourier transform of can be defined as: , the inverse transformation is ; This transform enables the formulation of operations such as filtering in the spectral domain;

[0062] Signal The filtering operation can be defined as:

[0063] ,

[0064] in, For signal, is the normalized Laplace matrix, is the eigenvector, is the corresponding eigenvalue, is the eigenvalue filter function, is a filter function based on the Laplace matrix.

[0065] Direct calculation is difficult because the eigenvalue decomposition of large Laplace matrices is very time-consuming. Therefore, spectral graph neural networks often use some polynomials instead. :

[0066] ,

[0067] in, is the eigenvalue filter function, is usually a learnable or fixed scalar, Various forms of polynomial bases are commonly used, is the corresponding eigenvalue, is the level of the polynomial basis.

[0068] It should also be noted that current multimodal recommendation models are all based on spatial graph neural networks, which have been shown to be low-pass filters that rely on a combination of monomial bases and fixed coefficients and have limited ability to utilize graph signals on user-item interaction graphs. Studies have shown that high-frequency graph signals contribute to recommendations, and that the low-frequency and high-frequency signals in the training data interactions have a strong linear correlation with the low-frequency and high-frequency signals in the test data. Researchers have designed various graph signal filters to model user behavior information at different frequencies. However, these filters rely on specific filter functions and still lack flexible and unified spectral graph neural network functions for modeling behavioral information.

[0069] Furthermore, multimodal recommendation models based on graph neural networks often focus on the use of low-frequency behavioral information when integrating user behavior with multimodal information. They lack in-depth research on different-frequency graph signals when fusing these signals with multimodal information; this results in poor performance when processing complex user behaviors. Furthermore, user behavior information of different frequencies exhibits distinct characteristics in multimodal fusion, making it difficult to effectively improve recommendation performance through simple frequency superposition.

[0070] Therefore, the present invention proposes a recommendation model based on graph neural network using spectral polynomial filter, namely PolyRec model, which effectively unifies multimodal information with behavioral information of different frequencies.

[0071] S102: Encode and fuse text modality and visual modality information in the user data and the project data using a multimodal encoder to obtain a first feature;

[0072] Preferably, a pre-trained encoder is used to learn the original multimodal features in the project data to obtain preliminary features;

[0073] Specifically, multimodal information provides rich and meaningful content information for projects. For projects with multimodal features, the corresponding pre-trained encoder is used to learn its original multimodal features. ,in, ,and , and then transform the multimodal features of different dimensions into the same feature space to obtain the preliminary features , expressed as:

[0074] ,

[0075] in, 、 represents the trainable transformation matrix and bias vector, is the original multimodal feature.

[0076] Preferably, the preliminary features are fused with the initial behavioral features in the project data and denoised using a behavior-guided purifier to obtain preference-related modal features;

[0077] Specifically, in order to avoid the influence of noise information contained in multimodal information, a behavior-guided purifier is used to alleviate this problem; the preliminary features With the project The initial behavioral features of and separate:

[0078] ,

[0079] in, For preliminary features, and is a learnable parameter, represents the pivot product, is the initial behavioral characteristic, is the sigmoid nonlinear activation function.

[0080] Preferably, the preference-related modal features are input into the item-item view encoder to perform graph convolution to obtain semantic correlation signals of multimodal features in the items;

[0081] In the embodiment of the present application, a fully connected graph is constructed based on the original multimodal features in the project data. , use KNN algorithm to Perform sparsification to obtain a sparse graph; normalize the sparse graph to obtain an item-item similarity matrix; use a graph neural network to propagate messages on the item-item similarity matrix to obtain the modal features of the items; based on the modal features of the items, obtain the modal features of the users; connect the modal features of the users and the modal features of the items to obtain multimodal features;

[0082] Specifically, in order to capture the semantic correlation signals of multimodal information in the project, graph convolution is performed on the project-item graph, and the original adjacency matrix KNN sparsification is applied to reduce the computational cost of propagating modal features. Specifically, first, according to each original modal feature quantify item-item affinity by similarity, and construct a fully connected graph , to indicate modality Item-item affinity in OK The elements in the column are:

[0083] ,

[0084] in, Indicates that in modal Next Project and projects The similarity between For the project exist The representation under the modal, For the project exist Representation under modality;

[0085] Then only keep each item with the maximum Similarity of the edges to obtain the matrix ,right Normalization is performed to alleviate the gradient explosion problem, ,in, yes degree matrix of ;

[0086] Item-item similarity matrix using graph neural network Perform message propagation to obtain the modal embedding of the item ,in, For the project In modal The following expression;

[0087] Finally, the modal features of the user are obtained by aggregating the modal features of the interaction items, which is expressed as:

[0088] ,

[0089] in, Represents a user and projects Neighbors, For the project In modal The following expression, For users In modal The following expression;

[0090] The modal features are obtained by connecting the user modal features and the item modal features:

[0091] ,

[0092] in, is the modal feature, is the user modal feature, is the project modal feature.

[0093] Preferably, a behavior-aware fusion device is used to fuse features of different modalities to obtain a first feature;

[0094] Specifically, modal preferences are extracted from behavioral characteristics:

[0095] ,

[0096] in, is the modality preference, is the sigmoid nonlinear activation function, are trainable weight vectors and bias vectors.

[0097] All modalities have modal sharing and modality-specific features , and users have consistent attention to the modal shared features; the modal shared features are extracted through the attention mechanism, which can be expressed as:

[0098] ,

[0099] in, represents the attention vector, and denote the weight matrix and bias vector respectively, is the modal feature, is the modal shared feature, is the normalization function, is the hyperbolic tangent activation function;

[0100] Modal sharing features It can be expressed as:

[0101] ,

[0102] in, is the modal shared feature, is the modal shared feature, is the modal feature.

[0103] Subtracting modality-shared features to obtain modality-specific features , the modality-specific features and modality-shared features Adaptive fusion to obtain the first feature , expressed as:

[0104] ,

[0105] in, As the first feature, is the modal shared feature, is the mode-specific feature, represents the pivot product, is the modality preference, is the number of modes.

[0106] S104: Encode the user data using a behavior encoder to obtain a second feature;

[0107] Preferably, the user data is encoded and converted into a graph structure, and the graph structure is modeled using a unified spectrum graph neural network function to capture different frequency components in the user data and obtain the second feature;

[0108] Preferably, a graph signal filtering function is defined to extract specific frequency components in the user data through the graph signal filtering function. The graph signal filtering function is:

[0109] In the high frequency range is monotonically increasing, and ,in ;

[0110] In the low frequency range is monotonically increasing, and ,in ;

[0111] In the intermediate frequency range The inner is a nonlinear transformation.

[0112] It should be noted that the signal filtering function is in the intermediate frequency range The transformation is nonlinear, but the overall value is close to 0.

[0113] In the embodiments of this application, and Frequency interval used to divide graphic signals; Represents high-frequency signals, Represents low-frequency signals; function In the interval It is flexible to change, which is conducive to the use of signals of different frequencies on the graph. In the present invention, a third-order Jacobi polynomial function is used as . It can also be replaced by other functions that satisfy the above properties.

[0114] It should be noted that both collaborative signals and semantic signals can affect the performance of multimodal recommendation; however, the current multimodal model based on graph neural network only considers user collaborative information under low-frequency signals, while ignoring information under other frequency signals; some works try to use various polynomial bases to utilize signals of different frequencies on the graph, but their ability to utilize graph signals is still limited. This paper believes that the key lies in the construction of graph signal filtering function rather than the selection of polynomial basis; therefore, a unified spectral graph neural network function is proposed to model user behavior information.

[0115] Studies have found that the interacting low-frequency and high-frequency signals in the training data have a strong positive linear correlation with the low-frequency and high-frequency signals in the test data, while the intermediate-frequency signals do not have a strong linear correlation; therefore, the present invention proposes a graph signal filtering function .

[0116] In the embodiment of the present application, a polynomial graph signal filter function is used to replace the graph signal filter function; function By adjacency matrix The polynomial basis of ;

[0117] In the embodiment of the present application, the second feature is obtained using the formula:

[0118] ,

[0119] in, For the second feature, Represents the adjacency matrix No. The polynomial basis represents that it can adopt various forms of polynomial basis. In the present invention, the monomial basis is adopted. It is the learnable coefficients of the polynomial basis, is the behavior representation of layer 0, is the number of levels of the polynomial basis, is the graph signal filtering function;

[0120] In the embodiment of the present application, the randomly sampled adjacency matrix The eigenvalue interval of Some points on as training samples and input them into the filter function and the polynomial filter function Then the Euclidean distance between them is calculated as the loss function to adaptively learn the coefficients of the polynomial basis; in order to avoid interference between multiple objective optimizations, The pre-trained filter is expressed as:

[0121] ,

[0122] in, is the filter loss function, is the graph signal filtering function, is the target image signal filtering function.

[0123] S106: Input the first feature and the second feature into the multimodal alignment module for fusion, and output a personalized recommendation result;

[0124] It should be noted that the current multimodal models based on graph neural networks only consider low-frequency behavioral information when processing the fusion of multimodal information and behavioral information, and they lack in-depth research on behavioral information of different frequencies; however, directly aligning behavioral information of different frequencies with multimodal information does not produce the best results; by studying the relationship between behavioral information of different frequencies and multimodal information, it is found that these behavioral signals exhibit different characteristics in the fusion process, and they need to be adjusted to better match the multimodal information; therefore, a frequency signal scaler is proposed to adjust the behavioral information at different frequencies.

[0125] Preferably, a frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as:

[0126] ,

[0127] in, Represent the scaling functions of high-frequency, medium-frequency and low-frequency signals respectively, , .

[0128] The scaling function is used to make subtle adjustments to the behavior information of different frequencies, and the frequency signal scaler Filter function with the original graphic signal Multiply to get a new filter function = , the adjusted filter function is used as a supervisory signal to optimize the original polynomial basis, which is expressed as:

[0129] ,

[0130] in, The loss function for pre-training the filter function, is the polynomial filter function, is the adjusted filter function.

[0131] In the embodiment of the present application, the first feature and the second feature are fused to obtain the final representation, which is expressed as follows: .

[0132] In the embodiment of the present application, in order to promote the second feature and the first feature Based on the exploration, a self-supervision task was designed. The mathematical expression of the supervision task is expressed as:

[0133] ,

[0134] in, is the temperature hyperparameter of the softmax normalization function, For the project behavior indicates that Negative samples behavior indicates that For users The behavioral information indicates that For users The multimodal representation of Negative samples The multimodal representation of For the project The multimodal representation of Negative samples The multimodal representation of For user collection, A collection of items.

[0135] Then use the inner product to calculate the user and projects Similarity between:

[0136] ,

[0137] in, is the similarity, For users The final expression of For the project The final expression of .

[0138] During the model training phase, Bayesian Personalized Ranking (BPR) loss is used As a basic optimization task, assuming that users prefer items they have historically interacted with to items they have not clicked, we have:

[0139] ,

[0140] in, is a user The set of neighbors of For items, users, and negative sample items, For user collection, is the sigmoid nonlinear activation function, For users and projects The prediction score of For users and projects The prediction score of

[0141] The final loss function can be expressed as follows:

[0142] ,

[0143] in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, To control the hyperparameters of the contrast auxiliary tasks separately, is a hyperparameter of the regularization effect, The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

[0144] The above is a schematic scheme of a multimodal recommendation method based on a spectral polynomial filter in this embodiment. It should be noted that the technical scheme of the multimodal recommendation system based on a spectral polynomial filter and the technical scheme of the multimodal recommendation method based on a spectral polynomial filter are based on the same concept. For details not described in detail in the technical scheme of the multimodal recommendation system based on a spectral polynomial filter in this embodiment, please refer to the description of the technical scheme of the multimodal recommendation method based on a spectral polynomial filter.

[0145] The multimodal recommendation system based on the spectral polynomial filter in this embodiment includes:

[0146] Acquisition module, used to obtain user data and project data;

[0147] A feature acquisition module is used to input user data and project data into the PolyRec model, which includes a multimodal encoder, a behavioral encoder, and a multimodal alignment module; the multimodal encoder is used to encode and fuse the textual modality and visual modality information in the user data and project data to obtain the first feature; the behavioral encoder is used to encode the user data to obtain the second feature;

[0148] The feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion and output personalized recommendation results.

[0149] This embodiment further provides an electronic device suitable for multimodal recommendation based on a spectral polynomial filter, including:

[0150] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the multimodal recommendation method based on spectral polynomial filter as proposed in the above embodiment.

[0151] This embodiment further provides a storage medium having a computer program stored thereon. When the program is executed by a processor, the multimodal recommendation method based on the spectral polynomial filter proposed in the above embodiment is implemented.

[0152] The storage medium proposed in this embodiment and the multimodal recommendation method based on spectral polynomial filter proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0154] Example 2

[0155] Referring to Tables 1 to 3, an embodiment of the present invention provides a multimodal recommendation method based on a spectral polynomial filter. In order to verify its beneficial effects, comparative results of several schemes are provided.

[0156] To evaluate the performance of the PolyRec model, experiments were conducted on three widely used recommendation datasets: Baby, Sports, and Clothing. Detailed data for these datasets is shown in Table 1.

[0157] Table 1 Dataset statistics

[0158] ,

[0159] In Table 1, Datasets refers to the three commonly used experimental datasets on Amazon used in the experiment: Baby, Sports, and Clothing; Users and Items respectively represent the total number of users and items contained in these three datasets; Behaviors represents the total number of interaction data of users and items in these three datasets; Density represents the density of behavioral data in the three datasets.

[0160] The optimal parameter selection of the PolyRec model of the present invention on three data sets is shown in Table 2;

[0161] Table 2 Parameter selection

[0162] ,

[0163] In order to verify the technical effect of the present invention, two indicators are used to evaluate the model performance, namely Recall@K and NDCG@K, and the latest multimodal model is selected as the benchmark model. ,The specific recommended performance is shown in Table 3 below;

[0164] Table 3 Evaluation results

[0165] ,

[0166] In Table 3, MF is a matrix decomposition model that models the behavioral information of users and projects. LightGCN is a graph-based model for modeling behavioral information. JGCF is a spectrum-based model for modeling behavioral information. VBPR is a multimodal model that uses matrix decomposition to model behavioral information and then fuses it with multimodal information. MMGCN is a multimodal model that uses graphs to model behavioral information and then fuses it with multimodal information. CRCN is a multimodal model that eliminates noise information in behavioral information based on MMGCN and then combines it with multimodal information. SLMRec is a model that adds self-supervised learning technology to MMGCN. BM3 simplifies the self-supervised learning framework in SLMRec and directly performs data perturbation through dropout. MICRO takes into account the multimodal relationship between projects when modeling the multimodal information of projects, thereby enhancing the multimodal recommendation model. MGCN uses behavioral information to purify modal noise.

[0167] As shown in Table 3, experiments on three commonly used recommendation datasets demonstrate that the PolyRec model of the present invention can effectively model user behavior data and fully integrate user and item behavior characteristics with multimodal features, significantly improving the performance of current multimodal recommendation.

[0168] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal recommendation method based on spectral polynomial filter, characterized in that: include: Obtain user data and project data; the user data includes basic user information and user behavior information of different frequencies; Inputting the user data and the project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavior encoder, and a multimodal alignment module; Encoding and fusing text modality and visual modality information in the user data and the project data using the multimodal encoder to obtain a first feature; Encoding the user data using the behavior encoder to obtain a second feature; The first feature and the second feature are input into the multimodal alignment module for fusion, and a personalized recommendation result is output.

2. The multimodal recommendation method based on spectral polynomial filter according to claim 1, wherein: The multimodal encoder consists of a pre-trained encoder, a behavior-guided purifier, an item-item view encoder, and a behavior-aware fusion.

3. The multimodal recommendation method based on spectral polynomial filter according to claim 2, wherein: Encoding and fusing text modality and visual modality information in user and project data using the multimodal encoder to obtain a first feature includes: Using the pre-trained encoder to learn original multimodal features in the project data to obtain preliminary features; Fusing the preliminary features with initial behavioral features in the project data, and performing denoising processing using the behavioral guidance purifier to obtain preference-related modal features; Inputting the preference-related modal features into the item-item view encoder to perform graph convolution to obtain semantic correlation signals of multimodal features in the item; The behavior perception fuser is used to fuse features of different modalities to obtain a first feature.

4. The multimodal recommendation method based on spectral polynomial filter according to claim 3, wherein: Inputting the preference-related modal features into the item-item view encoder to perform graph convolution to obtain semantically relevant signals of multimodal features in the item includes: Based on the original multimodal features in the project data, a fully connected graph S is constructed m , use KNN algorithm to analyze S m Perform sparse processing to obtain a sparse graph; Normalizing the sparse graph to obtain an item-item similarity matrix, performing message propagation on the item-item similarity matrix using a graph neural network to obtain modal features of the items, and obtaining modal features of the users based on the modal features of the items; The modal features of the user and the modal features of the item are connected to obtain multimodal features.

5. The multimodal recommendation method based on spectral polynomial filter according to claim 4, characterized in that: Encoding the user data using the behavior encoder to obtain the second feature includes: Encoding the user data, converting the user data into a graph structure, modeling the graph structure using a unified spectral graph neural network function, capturing different frequency components in the user data, and obtaining a second feature; A graph signal filtering function is defined to extract specific frequency components from user data. The graph signal filtering function is: Monotonically increasing in the high-frequency interval [-1, α], and f(x)<0, where -1<α<0; Monotonically increasing in the low-frequency interval [β, 1], and f(x)>0, where 0<β<1; The transformation is nonlinear in the intermediate frequency interval [α, β].

6. The multimodal recommendation method based on spectral polynomial filter according to claim 5, wherein: The second characteristics include: The frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as: Among them, g1(x), g2(x), and g3(x) represent the scaling functions of high-frequency, medium-frequency, and low-frequency signals, respectively, and -1<α<0, 0<β<1.

7. The multimodal recommendation method based on spectral polynomial filter according to claim 6, wherein: PolyRec models include: A loss function is designed to optimize the PolyRec model. The loss function is expressed as: in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, λ1 is the hyperparameter for controlling the contrast auxiliary tasks, and λ2 is the hyperparameter for the regularization effect. The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

8. A system using the multimodal recommendation method based on spectral polynomial filter according to any one of claims 1 to 7, characterized in that: include: Acquisition module, used to obtain user data and project data; a feature acquisition module, configured to input the user data and project data into a PolyRec model, the PolyRec model comprising a multimodal encoder, a behavioral encoder, and a multimodal alignment module; and utilize the multimodal encoder to encode and fuse textual modality and visual modality information in the user data and project data to obtain a first feature; Encoding the user data using the behavior encoder to obtain a second feature; The feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion, and output a personalized recommendation result.

9. An electronic device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal recommendation method based on spectral polynomial filter according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the multimodal recommendation method based on spectral polynomial filter according to any one of claims 1 to 7.

Images