Multimodal recommendation method and system based on spectrogram polynomial filter

Abstract

The invention relates to the technical field of data processing, and discloses a multimodal recommendation method and system based on a spectrogram polynomial filter.The method comprises the steps that user data and project data are input into a PolyRec model, and the PolyRec model comprises a multimodal encoder, a behavior encoder and a multimodal alignment module; encoding and fusing the user data and the project data by using a multi-mode encoder to obtain a first feature; encoding the user data by using a behavior encoder to obtain a second feature; and inputting the first feature and the second feature into a multi-modal alignment module for fusion, and outputting a personalized recommendation result. According to the method, the multi-modal characteristics of the user and the behavior information of different frequencies are integrated, so that the user preferences can be captured more comprehensively; and a frequency signal scaler and a unified spectrogram neural network function are adopted, so that the model can flexibly adapt to the characteristics of different frequency signals, and the robustness and adaptability of the recommendation system are further enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a multimodal recommendation method and system based on a spectrogram polynomial filter. Background Art

[0002] In the digital age, recommender systems have become a core component of e-commerce and content delivery platforms, which enhance user experience by analyzing user behavior and providing customized content. These systems particularly benefit from the fusion of multimodal information, such as images, text, and user interaction data, which makes recommendations more accurate and personalized. Early work on multimodal recommender systems enhanced traditional matrix factorization methods by integrating visual features, while in recent years, graph neural networks (GNNs) have been widely used in recommender systems due to their strong ability to process graph-structured data, especially in collaborative filtering-based recommendations. These graph neural network models are able to merge multimodal information from different views and learn complex representations of user preferences.

[0003] Although multimodal recommendation methods based on graph neural networks have achieved certain success, they still face challenges in processing the integration of user behavior information and multimodal information. First, existing models often ignore high-frequency signals in user-item interaction graphs, which are crucial for capturing users' immediate preference changes. Second, these models lack effective guidance when integrating multimodal information and behavioral information of different frequencies, resulting in poor performance when dealing with complex user behaviors. In addition, existing models rely on specific graph signal filters and lack flexibility and uniformity, which limits their potential in capturing user preferences. Therefore, developing a recommendation system that can comprehensively consider multimodal information and user behavior information of different frequencies is of great significance to improving recommendation quality and user satisfaction. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a multimodal recommendation method based on spectral polynomial filter to solve the problems of insufficient modeling of user behavior information of different frequencies and lack of effective guidance for the fusion of multimodal information and behavior information in the existing multimodal recommendation system.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a multimodal recommendation method based on a spectral polynomial filter, comprising: Get user data and project data; Inputting the user data and the project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavioral encoder, and a multimodal alignment module; Using the multimodal encoder to encode and fuse text modality and visual modality information in the user data and the project data to obtain a first feature; Encoding the user data using the behavior encoder to obtain a second feature; The first feature and the second feature are input into the multimodal alignment module for fusion, and a personalized recommendation result is output.

[0007] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter described in the present invention, the multimodal encoder includes a pre-trained encoder, a behavior-guided purifier, an item-item view encoder and a behavior-aware fusion.

[0008] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, wherein: using the multimodal encoder to encode and fuse the text modality and visual modality information in the user and project data, obtaining the first feature includes: Using the pre-trained encoder to learn original multimodal features in the project data to obtain preliminary features; The preliminary features are merged with the initial behavior features in the project data, and denoised using the behavior-guided purifier to obtain preference-related modal features; Inputting the preference-related modal features into the item-item view encoder to perform graph convolution to obtain semantically relevant signals of multimodal features in the item; The behavior perception fuser is used to fuse features of different modalities to obtain a first feature.

[0009] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, wherein: the preference-related modal features are input into the item-item view encoder to perform graph convolution, and the semantic related signals of the multimodal features in the item are obtained, including: Build a fully connected graph based on the raw multimodal features in the project data , using the KNN algorithm Perform sparse processing to obtain a sparse graph; Normalizing the sparse graph to obtain an item-item similarity matrix, using a graph neural network to perform message propagation on the item-item similarity matrix to obtain modal features of the item, and obtaining modal features of the user based on the modal features of the item; The modal features of the user and the modal features of the item are connected to obtain multimodal features.

[0010] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, wherein: encoding the user data by using the behavior encoder to obtain the second feature includes: Encoding the user data, converting the user data into a graph structure, modeling the graph structure using a unified spectrum graph neural network function, capturing different frequency components in the user data, and obtaining a second feature; A graph signal filtering function is defined, and a specific frequency component in the user data is extracted by the graph signal filtering function. The graph signal filtering function is: In the high frequency range is monotonically increasing, and ,in ; In the low frequency range is monotonically increasing, and ,in ; In the mid-frequency range The inside is a nonlinear transformation.

[0011] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter of the present invention, the second feature includes: The frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as: , in, Represent the scaling functions of high-frequency, medium-frequency and low-frequency signals respectively, , .

[0012] As a preferred solution of the multimodal recommendation method based on spectral polynomial filter described in the present invention, the PolyRec model includes: A loss function is designed to optimize the PolyRec model. The loss function is expressed as: , in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, To control the hyperparameters of the contrast auxiliary tasks separately, is a hyperparameter of the regularization effect, The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

[0013] In a second aspect, the present invention provides a multimodal recommendation system based on spectral polynomial filter, comprising: Acquisition module, used to obtain user data and project data; A feature acquisition module, used for inputting the user data and project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavioral encoder, and a multimodal alignment module; using the multimodal encoder to encode and fuse text modality and visual modality information in the user data and project data to obtain a first feature; using the behavioral encoder to encode the user data to obtain a second feature; A feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion, and output a personalized recommendation result.

[0014] In a third aspect, the present invention provides an electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal recommendation method based on the spectral polynomial filter are implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal recommendation method based on the spectral polynomial filter.

[0016] Compared with the prior art, the present invention has the following beneficial effects: the multimodal recommendation method based on spectral polynomial filter, namely the PolyRec model, proposed in the present invention improves the performance of the recommendation system in processing user behavior and multimodal information; by effectively integrating the multimodal features of users and behavioral information of different frequencies, the PolyRec model can more comprehensively capture user preferences and provide more accurate personalized recommendations; the use of frequency signal scalers and unified spectral graph neural network functions enables the model to flexibly adapt to the characteristics of signals of different frequencies, further enhancing the robustness and adaptability of the recommendation system. Overall, the present invention provides an efficient recommendation solution for e-commerce platforms, which can improve user experience and platform sales efficiency while protecting user privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 A schematic diagram of the overall process of a multimodal recommendation method based on a spectral polynomial filter according to an embodiment of the present invention; Figure 2 A schematic diagram of a PolyRec model of a multimodal recommendation method based on a spectral polynomial filter according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0020] Example 1 Reference Figure 1~Figure 2 , as an embodiment of the present invention, provides a multimodal recommendation method based on a spectral polynomial filter, comprising: S100: Obtain user data and project data; input the user data and project data into the PolyRec model, the PolyRec model includes a multimodal encoder, a behavior encoder and a multimodal alignment module; the PolyRec model is as follows Figure 2 As shown; In the embodiment of the present application, user data includes basic user information and user behavior information; for example: user ID, gender, historical behavior records, etc. Project data includes basic information of projects clicked by users, text modality and visual modality information of projects; for example: display pictures and text descriptions of projects, etc.

[0021] In the embodiment of the present application, firstly, an undirected graph is given ,set up and is the node set and edge set of the graph, and the user set is represented as , the set of items is represented as , the node set is represented as , the interaction matrix between users and items is , where if Users and Items have interactions, then . Adjacency Matrix It can be expressed as follows: , in, is the interaction matrix between users and items.

[0022] It should be noted that spectral graph neural networks perform graph convolution in the spectral domain of the Laplacian matrix. The normalized Laplacian matrix can be defined as: , in, represents the degree matrix of the graph, is the adjacency matrix, represents the identity matrix; Through eigenvalue decomposition, we can get its eigenvalue and corresponding eigenvector. ,in, is the eigenvector, is the corresponding eigenvalue, The characteristic value range is ; Signal The graphical Fourier transform of can be defined as: , the inverse transformation is ; This transform enables the formulation of operations such as filtering in the spectral domain; Signal The filtering operation can be defined as: , in, For signal, is the normalized Laplacian matrix, is the eigenvector, is the corresponding eigenvalue, is the eigenvalue filter function, is a filter function based on the Laplace matrix.

[0023] Direct calculation is difficult because the eigenvalue decomposition of large Laplacian matrices is very time-consuming, so spectral graph neural networks often use some polynomials instead. : , in, is the eigenvalue filter function, is usually a learnable or fixed scalar, Various forms of polynomial bases are commonly used, is the corresponding eigenvalue, is the level of the polynomial basis.

[0024] It should also be noted that current multimodal recommendation models are all based on spatial graph neural networks, which are proven to be a low-pass filter that relies on a combination of monomial bases with fixed coefficients and has limited ability to utilize graph signals on user-item interaction graphs; studies have shown that high-frequency graph signals are helpful for recommendations, and that low-frequency and high-frequency signals in training data interactions have a strong linear correlation with low-frequency and high-frequency signals in test data, and researchers have designed various graph signal filters to model user behavior information at different frequencies. However, these filters rely on specific filter functions and still lack flexible and unified spectral graph neural network functions for modeling behavioral information.

[0025] In addition, multimodal recommendation models based on graph neural networks tend to focus on the use of low-frequency behavior information when processing the fusion of user behavior and multimodal information. They lack in-depth research on different frequency graph signals when fusing these signals with multimodal information; this leads to poor performance of the model when processing complex user behaviors. Moreover, user behavior information of different frequencies presents different characteristics in multimodal fusion, which makes it difficult for simple frequency superposition to effectively improve recommendation performance.

[0026] Therefore, the present invention proposes a recommendation model based on graph neural network using spectral polynomial filter, namely, PolyRec model, which effectively unifies multimodal information with behavioral information of different frequencies.

[0027] S102: Encode and fuse text modality and visual modality information in the user data and the project data using a multimodal encoder to obtain a first feature; Preferably, the original multimodal features in the project data are learned using a pre-trained encoder to obtain preliminary features; Specifically, multimodal information provides rich and meaningful content information for projects. For projects with multimodal features, the corresponding pre-trained encoder is used to learn its original multimodal features. ,in, ,and , and then transform the multimodal features of different dimensions into the same feature space to obtain preliminary features , expressed as: , in, , represents the trainable transformation matrix and bias vector, is the original multimodal feature.

[0028] Preferably, the preliminary features are fused with the initial behavioral features in the project data, and denoised using a behavioral guidance purifier to obtain preference-related modal features; Specifically, in order to avoid the influence of noise information contained in multimodal information, a behavior-guided purifier is used to alleviate this problem; the preliminary features With Project The initial behavior features are combined together, and under the guidance of the initial behavior features, the preference-related modal features are and separate: , in, For preliminary characteristics, and is a learnable parameter, represents the pivot product, is the initial behavior characteristic, It is the sigmoid nonlinear activation function.

[0029] Preferably, the preference-related modal features are input into the item-item view encoder to perform graph convolution to obtain semantic correlation signals of multimodal features in the items; In the embodiment of the present application, a fully connected graph is constructed based on the original multimodal features in the project data. , using the KNN algorithm Perform sparsification processing to obtain a sparse graph; perform normalization processing on the sparse graph to obtain a project-project similarity matrix, use a graph neural network to propagate messages on the project-project similarity matrix to obtain the modal features of the project, and obtain the modal features of the user based on the modal features of the project; connect the modal features of the user and the modal features of the project to obtain multimodal features; Specifically, in order to capture the semantically relevant signals of multimodal information in the project, graph convolution is performed on the project-project graph. KNN sparsification is applied to reduce the computational cost of propagating modal features; specifically, first, according to each original modal feature quantify item-item affinity by similarity, and construct a fully connected graph , to indicate the modality Item-item affinity in OK The elements in the column are: , in, Indicates that in modal Next Project and Projects The similarity between For Project exist The representation under the modal, For Project exist Representation under modality; Then only keep each item with the maximum Similarity of edges to get the matrix ,right Normalize to alleviate the gradient explosion problem, ,in, yes The degree matrix of Using graph neural network to classify item-item similarity matrix Perform message propagation to obtain the modal embedding of the item ,in, For Project In modal The following expression; Finally, the modal features of the user are obtained by aggregating the modal features of the interaction items, which is expressed as: , in, Indicates user and Projects Neighbors, For Project In modal The following expression, For users In modal The following expression; The modal features are obtained by connecting the user modal features and the item modal features: , in, is the modal feature, is the user modality feature, It is the modal feature of the project.

[0030] Preferably, a behavior-aware fuser is used to fuse features of different modalities to obtain a first feature; Specifically, modal preferences are extracted from behavioral features: , in, is the modal preference, is the sigmoid nonlinear activation function, are trainable weight vectors and bias vectors.

[0031] All modalities have modal sharing and modality-specific features , and users have consistent attention to the modal shared features; the modal shared features are extracted through the attention mechanism, which can be expressed as: , in, represents the attention vector, and denote the weight matrix and bias vector respectively, is the modal feature, is the modal sharing feature, is the normalization function, is the hyperbolic tangent activation function; Modal Sharing Features It can be expressed as: , in, is the modal sharing feature, is the modal sharing feature, is a modal feature.

[0032] Subtract modality-shared features to obtain modality-specific features , the modality-specific features Shared features with modalities Adaptive fusion to obtain the first feature , expressed as: , in, As the first feature, is the modal sharing feature, is the mode-specific feature, represents the pivot product, is the modal preference, is the number of modes.

[0033] S104: Encode the user data using a behavior encoder to obtain a second feature; Preferably, the user data is encoded, the user data is converted into a graph structure, the graph structure is modeled using a unified spectrum graph neural network function, different frequency components in the user data are captured, and the second feature is obtained; Preferably, a graph signal filtering function is defined, and a specific frequency component in the user data is extracted by the graph signal filtering function. The graph signal filtering function is: In the high frequency range is monotonically increasing, and ,in ; In the low frequency range is monotonically increasing, and ,in ; In the mid-frequency range The inside is a nonlinear transformation.

[0034] It should be noted that the signal filtering function is in the intermediate frequency range The transformation is nonlinear, but the overall value is close to 0.

[0035] In the embodiments of the present application, and Frequency interval used to divide the graphic signal; represents high frequency signals, Represents low-frequency signal; function In the interval The third-order Jacobi polynomial function is used as . It can also be replaced by other functions that satisfy the above properties.

[0036] It should be noted that both collaborative signals and semantic signals can affect the performance of multimodal recommendation; however, the current multimodal model based on graph neural network only considers user collaborative information under low-frequency signals, but ignores information under other frequency signals; some works try to use various polynomial bases to utilize signals of different frequencies on the graph, but their ability to utilize graphic signals is still limited. The present invention believes that the key lies in the construction of the graph signal filter function rather than the selection of the polynomial basis; therefore, a unified spectral graph neural network function is proposed to model user behavior information.

[0037] Studies have found that the interacting low-frequency and high-frequency signals in the training data have a strong positive linear correlation with the low-frequency and high-frequency signals in the test data, while the intermediate-frequency signals do not have a strong linear correlation; therefore, the present invention proposes a graph signal filtering function .

[0038] In the embodiment of the present application, a polynomial graph signal filtering function is used to replace the graph signal filtering function; function From the adjacency matrix The polynomial basis of ; In the embodiment of the present application, the second feature is obtained by the formula: , in, For the second feature, Represents the adjacency matrix No. The polynomial basis represents that it can adopt various forms of polynomial basis. In the present invention, a monomial basis is adopted. It is The learnable coefficients of the polynomial basis, is the behavior representation of layer 0, is the level of the polynomial basis, is the image signal filtering function; In the embodiment of the present application, the adjacency matrix is ​​randomly sampled. The characteristic value interval of Some points on as training samples and input them into the filter function and the polynomial filter function Then the Euclidean distance between them is calculated as the loss function to adaptively learn the coefficients of the polynomial basis; in order to avoid interference between multiple objective optimizations, The pre-trained filter is expressed as: , in, is the filter loss function, is the image signal filtering function, is the target image signal filtering function.

[0039] S106: Input the first feature and the second feature into a multimodal alignment module for fusion, and output a personalized recommendation result; It should be noted that the current multimodal models based on graph neural networks only consider low-frequency behavioral information when processing the fusion of multimodal information and behavioral information, and they lack in-depth research on behavioral information of different frequencies; however, directly aligning behavioral information of different frequencies with multimodal information does not produce the best results; by studying the relationship between behavioral information of different frequencies and multimodal information, it is found that these behavioral signals exhibit different characteristics in the fusion process, and they need to be adjusted to better match multimodal information; therefore, a frequency signal scaler is proposed to adjust the behavioral information at different frequencies.

[0040] Preferably, a frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as: , in, Represent the scaling functions of high-frequency, medium-frequency and low-frequency signals respectively, , .

[0041] The scaling function is used to make subtle adjustments to the behavior information of different frequencies, scaling the frequency signal The filter function of the original image signal Multiply them together to get a new filter function = , the adjusted filter function is used as a supervisory signal to optimize the original polynomial basis, expressed as: , in, is the loss function for pre-training the filter function, is the polynomial filter function, is the adjusted filter function.

[0042] In the embodiment of the present application, the first feature and the second feature are fused to obtain the final representation, which is expressed by the formula: .

[0043] In the embodiment of the present application, in order to promote the second feature and the first feature Based on the exploration, a self-supervision task was designed. The mathematical expression of the supervision task is expressed as: , in, is the temperature hyperparameter of the softmax normalization function, For Project The behavior indicates that Negative samples The behavior indicates that For users The behavioral information indicates that For users The multimodal representation of Negative samples The multimodal representation of For Project The multimodal representation of Negative samples The multimodal representation of For user collection, A collection of items.

[0044] Then use the inner product to calculate the user and Projects The similarity between: , in, is the similarity, For users The final expression of For Project final expression of .

[0045] During the model training phase, the Bayesian Personalized Ranking (BPR) loss is used As a basic optimization task, assuming that users prefer items they have historically interacted with to items they have not clicked, we have: , in, Is a user The set of neighbors of are items, users, and negative sample items, For user collection, is the sigmoid nonlinear activation function, For users and Projects The prediction score of For users and Projects The prediction score of The final loss function can be expressed as follows: , in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, To control the hyperparameters of the contrast auxiliary tasks separately, is a hyperparameter of the regularization effect, The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

[0046] The above is a schematic scheme of a multimodal recommendation method based on a spectrogram polynomial filter in this embodiment. It should be noted that the technical scheme of the multimodal recommendation system based on the spectrogram polynomial filter and the technical scheme of the multimodal recommendation method based on the spectrogram polynomial filter mentioned above belong to the same concept. For details not described in detail in the technical scheme of the multimodal recommendation system based on the spectrogram polynomial filter in this embodiment, please refer to the description of the technical scheme of the multimodal recommendation method based on the spectrogram polynomial filter mentioned above.

[0047] The multimodal recommendation system based on the spectral polynomial filter in this embodiment includes: Acquisition module, used to obtain user data and project data; A feature acquisition module is used to input user data and project data into a PolyRec model, the PolyRec model includes a multimodal encoder, a behavioral encoder and a multimodal alignment module; the multimodal encoder is used to encode and fuse the text modality and visual modality information in the user data and project data to obtain a first feature; the behavioral encoder is used to encode the user data to obtain a second feature; The feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion, and output personalized recommendation results.

[0048] This embodiment further provides an electronic device, which is applicable to multimodal recommendation based on a spectrogram polynomial filter, and includes: Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the multimodal recommendation method based on spectral polynomial filter as proposed in the above embodiment.

[0049] This embodiment further provides a storage medium on which a computer program is stored. When the program is executed by a processor, the multimodal recommendation method based on a spectral polynomial filter as proposed in the above embodiment is implemented.

[0050] The storage medium proposed in this embodiment and the multimodal recommendation method based on spectral polynomial filter proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0051] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0052] Example 2 Referring to Tables 1 to 3, an embodiment of the present invention provides a multimodal recommendation method based on a spectral polynomial filter. In order to verify its beneficial effects, comparison results of several solutions are provided.

[0053] In order to evaluate the performance of the PolyRec model of the present invention, experiments are conducted on three widely used datasets for recommendation. The datasets are Baby, Sports, and Clothing. The detailed data of the datasets are shown in Table 1 below; Table 1 Dataset statistics , In Table 1, Datasets refers to the three commonly used experimental datasets on Amazon used in the experiment: Baby, Sports, and Clothing; Users and Items respectively represent the total number of users and items contained in the three datasets; Behaviors represents the total number of interaction data of users and items in the three datasets; Density represents the density of the behavioral data of the three datasets.

[0054] The optimal parameter selection of the PolyRec model of the present invention on three data sets is shown in Table 2; Table 2 Parameter selection , In order to verify the technical effect of the present invention, two indicators are used to evaluate the model performance, namely Recall@K and NDCG@K, and the latest multimodal model is selected as the benchmark model. ,The specific recommended performance is shown in Table 3 below; Table 3 Evaluation results , In Table 3, MF: is a matrix decomposition model that models the behavioral information of users and items; LightGCN: is a model that models behavioral information based on graphs; JGCF: is a model that models behavioral information based on spectrograms; VBPR: is a multimodal model that uses matrix decomposition to model behavioral information and then fuses it with multimodal information; MMGCN: is a multimodal model that uses graphs to model behavioral information and then fuses it with multimodal information; CRCN: is a multimodal model that eliminates noise information in behavioral information based on MMGCN and then combines it with multimodal information; SLMRec: adds self-supervised learning technology to MMGCN; BM3: simplifies the self-supervised learning framework in SLMRec and directly performs data perturbation through dropout; MICRO: takes into account the multimodal relationship between items when modeling the multimodal information of items, thereby enhancing the multimodal recommendation model; MGCN: uses behavioral information to purify modal noise.

[0055] As shown in Table 3, experiments on three commonly used recommendation datasets prove that the PolyRec model of the present invention can effectively model user behavior data and can fully integrate user and project behavior characteristics with multimodal features, significantly improving the performance of current multimodal recommendations.

[0056] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal recommendation method based on spectral polynomial filter, characterized in that: include: Get user data and project data; Inputting the user data and the project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavioral encoder, and a multimodal alignment module; Using the multimodal encoder to encode and fuse text modality and visual modality information in the user data and the project data to obtain a first feature; Encoding the user data using the behavior encoder to obtain a second feature; The first feature and the second feature are input into the multimodal alignment module for fusion, and a personalized recommendation result is output.

2. The multimodal recommendation method based on spectral polynomial filter according to claim 1, characterized in that: The multimodal encoder includes a pre-trained encoder, a behavior-guided purifier, an item-item view encoder, and a behavior-aware fusion.

3. The multimodal recommendation method based on spectral polynomial filter according to claim 2, characterized in that: Using the multimodal encoder to encode and fuse text modality and visual modality information in user and project data to obtain the first feature includes: Using the pre-trained encoder to learn original multimodal features in the project data to obtain preliminary features; The preliminary features are merged with the initial behavior features in the project data, and denoised using the behavior-guided purifier to obtain preference-related modal features; Inputting the preference-related modal features into the item-item view encoder to perform graph convolution to obtain semantically relevant signals of multimodal features in the item; The behavior perception fuser is used to fuse features of different modalities to obtain a first feature.

4. The multimodal recommendation method based on spectral polynomial filter according to claim 3, characterized in that: Inputting the preference-related modal features into the item-item view encoder to perform graph convolution, and obtaining semantically relevant signals of multimodal features in the item includes: Build a fully connected graph based on the raw multimodal features in the project data , using the KNN algorithm Perform sparse processing to obtain a sparse graph; Normalizing the sparse graph to obtain an item-item similarity matrix, using a graph neural network to perform message propagation on the item-item similarity matrix to obtain modal features of the item, and obtaining modal features of the user based on the modal features of the item; The modal features of the user and the modal features of the item are connected to obtain multimodal features.

5. The multimodal recommendation method based on spectral polynomial filter according to claim 4, characterized in that: Encoding the user data by using the behavior encoder to obtain the second feature includes: Encoding the user data, converting the user data into a graph structure, modeling the graph structure using a unified spectrum graph neural network function, capturing different frequency components in the user data, and obtaining a second feature; A graph signal filtering function is defined, and a specific frequency component in the user data is extracted by the graph signal filtering function. The graph signal filtering function is: In the high frequency range is monotonically increasing, and ,in ; In the low frequency range is monotonically increasing, and ,in ; In the mid-frequency range The inside is a nonlinear transformation.

6. The multimodal recommendation method based on spectral polynomial filter according to claim 5, characterized in that: The second features include: The frequency signal scaler is used to adjust the second feature at different frequencies, which is expressed as: ， in, Represent the scaling functions of high-frequency, medium-frequency and low-frequency signals respectively, , .

7. The multimodal recommendation method based on spectral polynomial filter according to claim 6, characterized in that: PolyRec models include: A loss function is designed to optimize the PolyRec model. The loss function is expressed as: ， in, is the loss function, is the Bayesian personalized ranking loss function, is the loss function for the fusion of multimodal features and behavioral features, To control the hyperparameters of the contrast auxiliary tasks separately, is a hyperparameter of the regularization effect, The user and item behavior representation vectors of the initialization layer when modeling the user and item behavior information.

8. A system using the multimodal recommendation method based on spectral polynomial filter as claimed in any one of claims 1 to 7, characterized in that: include: Acquisition module, used to obtain user data and project data; A feature acquisition module, used for inputting the user data and project data into a PolyRec model, wherein the PolyRec model includes a multimodal encoder, a behavior encoder, and a multimodal alignment module; using the multimodal encoder to encode and fuse text modality and visual modality information in the user data and project data to obtain a first feature; Encoding the user data using the behavior encoder to obtain a second feature; A feature fusion module is used to input the first feature and the second feature into the multimodal alignment module for fusion, and output a personalized recommendation result.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal recommendation method based on spectral polynomial filter according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal recommendation method based on spectral polynomial filter as described in any one of claims 1 to 7.

Images