Personalized recommendation method based on multi-modal feature denoising adaptive fusion

Abstract

The application discloses a personalized recommendation method based on multi-modal feature denoising adaptive fusion, which can be applied to the technical field of graph neural networks. The application obtains denoised multi-modal features by performing wavelet transform frequency domain denoising on multi-modal original features, and obtains frequency fusion features by fusing the denoised multi-modal features. Meanwhile, the normalized adjacency matrix is obtained by clipping the interaction graph of users and items based on node degree sensitivity. Then, the collaborative representation of users and items is constructed according to the normalized adjacency matrix, and the modal representation of users and items is constructed according to the denoised multi-modal features. Finally, the final fusion representation of user behavior patterns and modal features is generated according to the frequency fusion features, the collaborative representation of users and items and the modal representation of users and items. Item recommendation prediction is performed according to the final fusion representation, which can improve the accuracy of the item recommendation result and meet the personalized demand of the recommendation content in the related field.

Description

Technical Field

[0001] This application relates to the field of graph neural network technology, and in particular to a personalized recommendation method based on multimodal feature denoising and adaptive fusion. Background Technology

[0002] In related technologies, recommender systems have been widely applied in e-commerce, social media, content distribution, and other fields. Current recommendation methods rely on user history and item attributes, achieving personalized recommendations through collaborative filtering or content-based approaches. However, when faced with data sparsity issues, both recommendation accuracy and coverage are limited. Multimodal recommender systems (MRSs) introduce feature information from multiple modalities such as images and text, offering advantages in mining item appearance, semantic descriptions, and usage scenarios. They can more deeply characterize user preference features, thereby improving the accuracy and diversity of recommendation results. Among these, multimodal recommendation methods based on Graph Convolutional Networks (GCNs) are particularly effective at modeling high-order relationships between users and items and capturing complex interaction structures. However, current GCN-based recommendation methods introduce noise pollution during training, and feature fusion processes are often based on static or linear fusion strategies, leading to recommendations that may differ significantly from actual user needs.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a personalized recommendation method based on multimodal feature denoising and adaptive fusion, which can effectively meet the personalized needs of recommendation content in related fields and reduce the difference between recommendation results and actual needs.

[0005] To achieve the above objectives, one aspect of this application proposes a personalized recommendation method based on multimodal feature denoising and adaptive fusion, the method comprising the following steps:

[0006] Construct multimodal raw features and user-item interaction graphs, wherein the multimodal raw features include the raw features of items in different modalities; the different modalities include visual modalities or text modalities;

[0007] Wavelet transform is performed on the original multimodal features to obtain denoised multimodal features, and the denoised multimodal features are fused to obtain frequency fusion features;

[0008] The user-item interaction graph is cropped based on node degree sensitivity to obtain a normalized adjacency matrix.

[0009] Construct a collaborative representation of users and items based on the normalized adjacency matrix;

[0010] Based on the denoised multimodal features, construct modal representations of users and items;

[0011] The final fused representation of user behavior patterns and modal features is generated based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items.

[0012] Item recommendation prediction is performed based on the final fused representation.

[0013] In some embodiments, performing wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and fusing the denoised multimodal features to obtain frequency fusion features, includes:

[0014] The original multimodal features are spatially projected to obtain multimodal projected features;

[0015] Discrete wavelet transform is performed on the multimodal projection features to obtain the low-frequency and high-frequency components of different modes;

[0016] Calculate cross-modal frequency-aware similarity weights based on the low-frequency components and the high-frequency components;

[0017] The low-frequency component and the high-frequency component are fused in the frequency domain according to the frequency-aware similarity weight to obtain the frequency fusion feature.

[0018] In some embodiments, the formula for calculating the frequency fusion feature is as follows:

[0019] ;

[0020] In the formula, Indicates frequency fusion characteristics; Represents the discrete wavelet transform; The frequency-perceived similarity weights for low-frequency components; Frequency-perceived similarity weights representing high-frequency components; Represents the low-frequency approximation coefficients in the visual modality; Represents the low-frequency approximation coefficients in the text modality; This represents the high-frequency detail coefficients after denoising of the visual modality; Represents the high-frequency detail coefficients after text modality denoising; Indicates the execution level corresponding to the discrete wavelet transform; The basis functions represent the discrete wavelet transform.

[0021] In some embodiments, the formula for calculating the frequency-aware similarity weight is as follows:

[0022] ;

[0023] In the formula, This represents the frequency-aware similarity weight; , , Indicates low-frequency components; Indicates high-frequency components; This represents a non-linear activation function.

[0024] In some embodiments, the step of cropping the user-item interaction graph based on node degree sensitivity to obtain a normalized adjacency matrix includes:

[0025] Obtain the degree of each node in the user-item interaction graph;

[0026] The sampling weight is calculated based on the degree of the node;

[0027] Obtain the number of edges in the user-item interaction graph and obtain the preset pruning ratio;

[0028] The number of sampling edges is determined based on the number of edges and the preset pruning ratio;

[0029] The user-item interaction graph is pruned according to the normalized sampling weights to obtain the target edge set corresponding to the number of sampling edges;

[0030] Construct a pruned adjacency matrix based on the target edge set;

[0031] The pruned adjacency matrix is ​​normalized to obtain the normalized adjacency matrix.

[0032] In some embodiments, the expression for the normalized adjacency matrix is ​​as follows:

[0033] ;

[0034] In the formula, Represents the normalized adjacency matrix; Represents the adjacency matrix after pruning; Represents the adjacency matrix after pruning The degree matrix.

[0035] In some embodiments, constructing a collaborative representation of users and items based on the normalized adjacency matrix includes:

[0036] Initialize the ID embedding matrix;

[0037] Feature propagation rules are constructed based on the normalized adjacency matrix;

[0038] Based on the initialized ID embedding matrix and the feature propagation rule, the features of each layer in the user-item interaction graph are aggregated to obtain the collaborative representation of the user and items.

[0039] In some embodiments, constructing modal representations of users and items based on the denoised multimodal features includes:

[0040] Calculate the cosine similarity between items in each modality;

[0041] Construct a sparse post-similarity matrix based on the cosine similarity between the items;

[0042] The sparsed similarity matrix is ​​normalized to obtain the normalized similarity matrix.

[0043] The item modal representation is constructed based on the normalized similarity matrix and the denoised multimodal features;

[0044] Construct a user modality representation based on the item modality representation;

[0045] The modal representations of the item and the user are concatenated to obtain the modal representations of the user and the item.

[0046] In some embodiments, generating a final fused representation of user behavior patterns and modal features based on the frequency fusion features, the user-item collaborative representation, and the user-item modal representation includes:

[0047] The frequency fusion features are injected into the collaborative representation of the user and the item to obtain the injected collaborative representation;

[0048] Calculate cross-modal features based on the injected collaborative representation and the modal representations of users and items;

[0049] Adaptive weights are calculated based on the injected collaborative representation, the modal representations of the user and the item, and dynamic parameters, wherein the dynamic parameters are adjusted based on the reliability of the collaborative signal;

[0050] The final fused representation of user behavior patterns and modal features is calculated based on the adaptive weights, the modal representations of users and items, and the collaborative representations of users and items.

[0051] In some embodiments, the step of predicting item recommendations based on the final fused representation includes:

[0052] The final user representation and the final item representation are calculated based on the collaborative representation of the user and the item and the final fused representation, respectively.

[0053] The recommendation prediction score is determined based on the inner product of the user's final representation and the item's final representation;

[0054] Item recommendations are made based on the predicted recommendation scores.

[0055] To achieve the above objectives, another aspect of this application proposes a personalized recommendation system based on multimodal feature denoising and adaptive fusion, the system comprising:

[0056] The first module is used to construct multimodal raw features and user-item interaction graphs. The multimodal raw features include the raw features of items in different modalities. The different modalities include visual modalities or text modalities.

[0057] The second module is used to perform wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and to fuse the denoised multimodal features to obtain frequency fusion features.

[0058] The third module is used to trim the user-item interaction graph based on node degree sensitivity to obtain a normalized adjacency matrix.

[0059] The fourth module is used to construct a collaborative representation of users and items based on the normalized adjacency matrix;

[0060] The fifth module is used to construct modal representations of users and items based on the denoised multimodal features;

[0061] The sixth module is used to generate a final fused representation of user behavior patterns and modal features based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items;

[0062] The seventh module is used to perform item recommendation prediction based on the final fused representation.

[0063] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0064] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0065] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0066] The embodiments of this application include at least the following beneficial effects: This application provides a personalized recommendation method based on multimodal feature denoising and adaptive fusion. This method constructs multimodal original features of items in different modalities and user-item interaction graphs. Then, it performs wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and fuses these denoised multimodal features to obtain frequency fusion features. This reduces the impact of noise on the recommendation process. Simultaneously, based on node degree sensitivity, it trims the user-item interaction graph to obtain a normalized adjacency matrix, thereby purifying the interaction graph and improving recommendation accuracy. Then, it constructs a collaborative representation of users and items based on the normalized adjacency matrix, and constructs modal representations of users and items based on the denoised multimodal features. Finally, it generates a final fusion representation of user behavior patterns and modal features based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items. Finally, it performs item recommendation prediction based on the final fusion representation. This effectively improves the accuracy of item recommendation results, effectively meets the personalized needs of recommendation content in related fields, and reduces the difference between recommendation results and actual needs. Attached Figure Description

[0067] Figure 1 This is a flowchart of a personalized recommendation method based on multimodal feature denoising and adaptive fusion provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of a personalized recommendation model based on multimodal feature denoising and adaptive fusion provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application.

[0070] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0071] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0073] In related technologies, current recommendation methods often contain noise information unrelated to users' true preferences, whether extracted through pre-trained models or from other external data sources. In product recommendation scenarios, complex backgrounds in clothing images (such as non-subject elements like floral decorations) can interfere with the extraction of core visual features like fabric and cut; excessive marketing embellishments in product descriptions (such as "celebrity-endorsed") can obscure their true attributes. This noise accumulates and amplifies during graph-based information propagation, leading to a decline in the quality of user and item representations and thus affecting recommendation accuracy. Furthermore, existing recommendation methods often employ static or linear fusion strategies, making it difficult to adaptively adjust based on differences in modal preferences when users interact with different types of items. For example, users buying cosmetics tend to focus more on textual features like ingredient lists and user reviews, while furniture buyers rely more on visual features like product images and design drawings. Models lacking dynamic adjustment mechanisms for modal importance often fail to achieve optimal fusion results when faced with changes in user preferences or noise in features, leading to decreased recommendation performance.

[0074] In view of this, this application provides a personalized recommendation method based on multimodal feature denoising and adaptive fusion, which can effectively meet the personalized needs of recommended content in related fields and reduce the difference between the recommendation results and actual needs.

[0075] The personalized recommendation method based on multimodal feature denoising and adaptive fusion provided in this application relates to the field of graph neural network technology. This personalized recommendation method based on multimodal feature denoising and adaptive fusion can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the personalized recommendation method based on multimodal feature denoising and adaptive fusion, but is not limited to the above forms.

[0076] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0078] The embodiments of this application will be described in detail below with reference to the accompanying drawings:

[0079] Figure 1 This is an optional flowchart of a personalized recommendation method based on multimodal feature denoising and adaptive fusion provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S170:

[0080] Step S110: Construct multimodal raw features and user-item interaction graphs, wherein the multimodal raw features include the raw features of the item in different modalities; different modalities include visual modalities or text modalities;

[0081] Step S120: Perform wavelet transform on the original multimodal features to obtain denoised multimodal features, and fuse the denoised multimodal features to obtain frequency fusion features;

[0082] Step S130: Based on node degree sensitivity, the interaction graph between users and items is cropped to obtain a normalized adjacency matrix;

[0083] Step S140: Construct a collaborative representation of users and items based on the normalized adjacency matrix;

[0084] Step S150: Construct modal representations of users and items based on the denoised multimodal features;

[0085] Step S160: Generate the final fused representation of user behavior patterns and modal features based on frequency fusion features, collaborative representation of users and items, and modal representation of users and items;

[0086] Step S170: Perform item recommendation prediction based on the final fused representation.

[0087] It is understandable that, in the process of constructing the multimodal original features and the user-item interaction graph in this embodiment, the user set is denoted as... The collection of items is Where M and N represent the total number of users and items, respectively. The embedding of user IDs and item IDs is represented by... and This indicates that d represents the shared embedding dimension. Each item i contains features from multiple modalities. ,in v represents the visual modality, and t represents the external text modality. Representing feature dimensions. A user's historical behavior is represented by an interaction matrix. It means that if This indicates that there is an interaction between user u and item i. All interactions form a bipartite graph. Its node set edge set The embodiments of this application learn a prediction function. It can fuse multimodal features to generate a preference prediction score for each user-item pair (u,i). .

[0088] It is understandable that, since the original multimodal features include complementary information, they inevitably carry inherent noise. Therefore, this embodiment, after obtaining the original multimodal features, performs wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and then fuses the denoised multimodal features to obtain frequency fused features. Wavelet transform-based cross-modal feature denoising and fusion (WFD) can perform denoising and fusion on features in the wavelet domain, thereby adaptively preserving effective signals at different frequencies. Compared with traditional single-modal denoising methods, the method in this embodiment can effectively reduce multimodal noise while preserving necessary semantic information.

[0089] Specifically, in this embodiment, during wavelet transform denoising and fusion, after spatially projecting the original multimodal features to obtain multimodal projected features, discrete wavelet transform is performed on the multimodal projected features to obtain low-frequency and high-frequency components of different modes. Cross-modal frequency-aware similarity weights are calculated based on the low-frequency and high-frequency components, and then frequency-domain fusion is performed on the low-frequency and high-frequency components based on the frequency-aware similarity weights to obtain the frequency-fused features. The original multimodal features can be projected into a shared latent space using the following formula:

[0090] ;

[0091] In the formula, Represents multimodal projection features; These are the original features of item i in mode m. and All of these are learnable parameters.

[0092] Then, for each multimodal projection feature... Perform L-level Discrete Wavelet Transform (DWT) while using Haar wavelets as basis functions. :

[0093] ;

[0094] In the formula, Represents the low-frequency approximation coefficients. Represents high-frequency detail coefficients; DWT recursively applies a low-pass filter g and a high-pass filter h to the coefficients of the (l-1)th layer and performs a 2x downsampling. To obtain the coefficients of the l-th layer:

[0095] ;

[0096] ;

[0097] In the formula, * represents the convolution operation.

[0098] To strike a balance between simplicity and efficiency, this embodiment selects the Haar wavelet and sets L=1. To distinguish between effective modes and noise in high-frequency components, a soft threshold is applied to the detail coefficients based on Donoho's contraction theory to obtain denoised coefficients. :

[0099] ;

[0100] In the formula, This represents the coefficients after denoising. Represents high-frequency detail coefficients. This represents the threshold parameter that controls the intensity of noise reduction.

[0101] Since the contributions of different modes may vary across different frequency bands, this embodiment proposes a frequency-aware fusion strategy. First, the cross-modal frequency-aware similarity weights are calculated:

[0102] ;

[0103] In the formula, This represents the frequency-aware similarity weight; , , Indicates low-frequency components. Represents high-frequency components. This represents a non-linear activation function.

[0104] Then, based on these weights, this embodiment uses inverse discrete wavelet transform (IDWT) to perform frequency domain fusion on the weighted coefficients to obtain the frequency fusion feature F:

[0105] ;

[0106] In the formula, Indicates frequency fusion characteristics; Represents the discrete wavelet transform; The frequency-perceived similarity weights for low-frequency components; Frequency-perceived similarity weights representing high-frequency components; Represents the low-frequency approximation coefficients in the visual modality; Represents the low-frequency approximation coefficients in the text modality; This represents the high-frequency detail coefficients after denoising of the visual modality; Represents the high-frequency detail coefficients after text modality denoising; Indicates the execution level corresponding to the discrete wavelet transform; The basis functions represent the discrete wavelet transform.

[0107] Through the above processing, complementary information can be preserved while suppressing noise, thus achieving the technical effect of providing reliable feature representation for recommendation tasks.

[0108] Understandably, although wavelet fusion denoising solves the noise problem at the feature level, noise may still propagate through the interaction graph during the recommendation process, thus affecting the accuracy of the recommendation results. Therefore, this embodiment prunes the user-item interaction graph based on node degree sensitivity. Specifically, this embodiment strategically prunes noisy connections in the interaction graph based on node connectivity (degree), while retaining key interaction information, to effectively prevent noise propagation during message passing and enhance the robustness of the recommendation model.

[0109] In this embodiment, after obtaining the degree of each node in the user-item interaction graph, the sampling weight is calculated based on the degree of the node; at the same time, the number of edges in the user-item interaction graph and the preset pruning ratio are obtained; then the number of sampling edges is determined based on the number of edges and the preset pruning ratio; then, the user-item interaction graph is pruned according to the normalized sampling weight to obtain the target edge set corresponding to the number of sampling edges; then, after constructing the pruned adjacency matrix based on the target edge set, the pruned adjacency matrix is ​​normalized to obtain the normalized adjacency matrix.

[0110] Specifically, given an interaction graph between a user and an item. in A set of nodes representing users and items. This represents the set of interactive edges. This embodiment constructs a symmetric adjacency matrix:

[0111] ;

[0112] In the formula, This represents the interaction matrix between users and items. For each edge in the interaction graph... And calculate the sampling weight based on the degree of the node. ,in, and represents the degree of node i and node j, respectively. This sampling weight ensures that edges connected to nodes with high elevations have a low retention probability, thus preventing excessive information aggregation.

[0113] In each training round, this embodiment uses a multinomial distribution (whose probability is determined by normalized weights). (Decision) Sampling Wherein, represents the preset pruning ratio. Then, based on the preset pruning ratio, the interaction graph is pruned to obtain the target edge set. A sparse adjacency matrix is ​​then constructed based on this target edge set as the pruned adjacency matrix. The pruned adjacency matrix is ​​then symmetrically normalized to obtain the normalized adjacency matrix:

[0114] ;

[0115] In the formula, Represents the normalized adjacency matrix; Represents the adjacency matrix after pruning; Represents the adjacency matrix after pruning The degree matrix.

[0116] The processing in this embodiment can minimize the spectral distance between the Laplacian matrix of the original interaction graph and the sparsed interaction graph while maintaining the structural integrity of the interaction graph.

[0117] During inference, this embodiment uses a fully normalized (unpruned) adjacency matrix for prediction. This mechanism of dynamic pruning during training effectively reduces the computational complexity of the training process while preserving the basic structural properties of the graph, achieving a balance between efficiency and performance.

[0118] Understandably, after denoising the original features and interaction graph structure, this embodiment creates a comprehensive feature representation that can simultaneously capture collaborative filtering patterns and content semantics. Based on this, this embodiment proposes a dual-view method that constructs user-item interactions through a bipartite graph and utilizes semantic similarity through intramodal isomorphic graphs to provide information-rich feature representations for subsequent multimodal fusion.

[0119] Specifically, this embodiment constructs a collaborative representation of users and items based on the normalized adjacency matrix. Specifically, after initializing the ID embedding matrix, a feature propagation rule is constructed based on the normalized adjacency matrix. Then, features from each layer of the user-item interaction graph are aggregated based on the initialized ID embedding matrix and the feature propagation rule to obtain the collaborative representation of users and items. It can be understood that, in this embodiment, to effectively capture the collaborative signals in user preferences and item relationships within the bipartite graph convolutional view, this invention designs a bipartite graph convolutional layer based on the aforementioned denoised graph structure to model higher-order collaborative relationships. First, the ID embedding matrix is ​​initialized. The following feature propagation rules are designed:

[0120] ;

[0121] In the formula, This represents the node representation of the p-th layer; It is the normalized adjacency matrix.

[0122] To utilize the collaborative information across all levels, this embodiment aggregates the features from each level using the following formula:

[0123] ;

[0124] In the formula, P represents the number of convolutional layers; It represents a collaborative representation of users and items, effectively aggregating neighborhood information from the denoised graph structure.

[0125] Understandably, in the isomorphic graph convolutional view, this embodiment constructs user and item modal representations based on denoised multimodal features. Specifically, this involves calculating the cosine similarity between items in each modality, constructing a sparse similarity matrix based on the cosine similarity, normalizing the sparse similarity matrix to obtain a normalized similarity matrix, constructing item modal representations based on the normalized similarity matrix and denoised multimodal features, constructing user modal representations based on the item modal representations, and concatenating the item and user modal representations to obtain the user and item modal representations. Specifically, in this embodiment, the semantic associations between items in the multimodal features can provide supplementary information for recommendation accuracy. This embodiment proposes a sparsification strategy based on K-Nearest Neighbors (KNN) to address the high computational cost and potential noise issues that may arise in dense item similarity maps. This embodiment, through each modality... First, calculate the cosine similarity between the items:

[0126] ;

[0127] In the formula, This represents the cosine similarity between item a and item b in mode m. This represents the characteristics of item a in mode m; This represents the characteristics of item b in mode m.

[0128] After calculating the pre-similarity between items, this embodiment retains the top K most similar neighbors of each item to sparse the corresponding interaction graph using the following formula:

[0129] ;

[0130] In the formula, This represents the cosine similarity between item a and item c in mode m after sparsification. This represents the cosine similarity between item a and item c in mode m. This represents the K nearest neighbors of an item in mode m.

[0131] Construct a sparsed similarity matrix based on all the sparsed cosine similarities. .

[0132] To balance the influence of nodes and determine training stability, this embodiment sparses the similarity matrix. Performing symmetric normalization, we obtain the normalized similarity matrix:

[0133] ;

[0134] Since semantic similarity decays rapidly with propagation distance, this embodiment employs a shallow graph convolutional structure for information propagation, thereby obtaining the item modality representation:

[0135] ;

[0136] In the formula, Represents the modal representation of an item; This indicates that the item ID is embedded in the denoised multimodal features through a gating mechanism. The characteristics of an item formed by combining elements.

[0137] Then, user modality representations are constructed by aggregating multimodal features of items that the user has interacted with:

[0138] ;

[0139] In the formula, Representing user modal representation; This represents the set of items that user u has interacted with; This represents the set of all users who have interacted with item i; the normalization process is to prevent the embedding scale from being too large.

[0140] This embodiment obtains the modal representation of the item. and user modal representation Then, the modal representation of the item is... and user modal representation By splicing the data together, a modal representation of the user and the item can be obtained. .

[0141] Understandably, based on the rich collaborative and semantic information captured by the aforementioned dual-view representation, this embodiment provides a behavior-guided frequency-aware fusion (BFF) approach. This approach dynamically adjusts the importance of each modality based on wavelet-refined features and user behavior patterns, thereby achieving adaptive personalized cross-modal fusion for each user. Based on this, this embodiment first provides a frequency-aware modulation method to incorporate frequency domain characteristics into the ID embedding. Specifically, this embodiment obtains an injected collaborative representation by injecting frequency fusion features into the collaborative representation of users and items. Then, cross-modal features are calculated based on the injected collaborative representation and the modal representations of users and items. Next, adaptive weights are calculated based on the injected collaborative representation, the modal representations of users and items, and dynamic parameters, where the dynamic parameters are adjusted based on the reliability of the collaborative signal. Finally, the final fused representation of user behavior patterns and modal features is calculated based on the adaptive weights, the modal representations of users and items, and the collaborative representation of users and items.

[0142] It is understood that the process of injecting frequency fusion features into the collaborative representation of users and items in this embodiment is as follows:

[0143] ;

[0144] In the formula, Indicates injection of collaborative representation; and Indicates learnable parameters; Represents the collaborative representation between users and items; This indicates element-wise multiplication.

[0145] Based on this enhanced injection collaborative representation, this embodiment provides a dual-path fusion mechanism to capture modality-specific features and cross-modality correlations.

[0146] The first path aims to learn cross-modal features guided by ID embeddings:

[0147] ;

[0148] In the formula, Indicates cross-modal features; and Indicates learnable parameters; Indicates injection of collaborative representation; Modal representation of users and items; This represents element-wise multiplication; Represents the Sigmoid function

[0149] The second path involves using an adaptive weight. This is used to integrate attention and interaction from different modalities. Among these, adaptive weights... By a dynamic parameter Perform flattening, dynamic parameters The adjustment is based on the reliability of the coordinated signal:

[0150] ;

[0151] In the formula, Indicates injection of collaborative representation; This represents the modal representation of users and objects in a visual modality. This represents the modal representation of users and items in the text modality; Indicates learnable parameters; This represents the transpose of the matrix corresponding to the modal representation of users and items in the visual modality. This represents the transpose of the matrix corresponding to the modal representation of users and items in the text modality.

[0152] This embodiment obtains adaptive weights. Then, by combining the modal representations of users and items and the collaborative representations of users and items, the final fused representation of user behavior patterns and modal features is calculated:

[0153] ;

[0154] In the formula, The final fused representation of user behavior patterns and modal features; Represents cross-modal features within the visual modality; This represents cross-modal features within the text modality.

[0155] This embodiment automatically adjusts the influence based on the consistency between multimodal signals and collaborative signals, as well as the reliability of user-item interactions, thereby achieving refined preference modeling that can adapt to individual users. To further enhance the consistency between collaborative information and multimodal information, this embodiment also introduces a contrastive learning objective function. Among them, the contrast loss on the user side. The following formula:

[0156] ;

[0157] Comparative loss on the item side The following formula:

[0158] ;

[0159] In the formula, It represents dynamic temperature parameters that can be adjusted according to batch size to encourage consistency in representation across different information sources; The final fusion representation on the user side corresponds to the transpose of the matrix. The node representation on the user's (u) side; The final fusion representation of the item side represents the transpose of the corresponding matrix; The node representation of item i; This represents the users in user set U and all users other than user u. This represents the items in item set I, excluding item i. Indicates user Side node representation and Represents items The nodes on the side are represented.

[0160] It is understood that this embodiment optimizes the model and recommends items based on Bayesian personalized ranking after obtaining the final fused representation. Specifically, this embodiment calculates the final user representation and the final item representation based on the collaborative representation of the user and the item, and the final fused representation, respectively. Then, it determines the recommendation prediction score based on the inner product of the final user representation and the final item representation, and then recommends items based on the recommendation prediction score. The final user representation is given by the following formula:

[0161] ;

[0162] In the formula, This represents the user's final opinion; This represents the collaborative representation on the user side. This represents the final fusion representation on the user side.

[0163] The item is ultimately represented by the following formula:

[0164] ;

[0165] In the formula, Indicates the final representation of an item; Represents the collaborative representation on the item side; This represents the final fusion representation of the item side.

[0166] To predict the probability of interaction, this embodiment obtains the predicted score by calculating the inner product of the user and the final representation of the item: .

[0167] This embodiment recommends items based on predicted scores, thereby making the recommended items more in line with the user's current needs, effectively meeting the personalized needs of recommended content in related fields, and reducing the difference between the recommendation results and actual needs.

[0168] It is understandable that this embodiment employs a combined loss function for model optimization training during the recommendation model optimization phase. This loss function includes Bayesian Personalized Ranking (BPR) loss, embedding regularization term, and the previously mentioned contrastive loss. The BPR loss is calculated as follows:

[0169] ;

[0170] The final overall objective function used for optimization is expressed by the following formula:

[0171] ;

[0172] In the formula, Let represent the training set of triples consisting of user u, positive sample item i that user u has interacted with, and negative sample item j that user u has not interacted with; Represents the Sigmoid function; Indicates the contrast learning loss; Indicates the application of model parameters Regularization term, used to prevent overfitting; hyperparameters and These are used to control the importance of contrastive learning and regularization terms, respectively.

[0173] In some embodiments, when applying the methods of this application, the data definitions and symbols are first explained as follows:

[0174] User and item collection: The collection of items is denoted as .

[0175] Interaction Matrix: Interaction Matrix , among which, element This indicates that there is an interaction between user u and item i. This indicates that there is no interaction between user u and item i.

[0176] Bipartite graph adjacency matrix: The row / column indices of adjacency matrix A correspond to user and item nodes, and its structure is as follows: The top left and bottom right matrices are zero matrices, and the top right matrices are interaction matrices. The lower left is the transpose of the interaction matrix. .

[0177] Denoising Feature Matrix: The visual and textual modal features of the item, after denoising, are represented as follows: and Each row of each matrix corresponds to a numerical representation of an item across d=8 semantic dimensions.

[0178] Based on the above definition, the method of this embodiment includes the following steps in application:

[0179] Step 1: Multimodal Feature and Interaction Graph Modeling. In this embodiment, the user-item interaction relationship is defined by the interaction matrix R. The adjacency matrix A of the bipartite graph constructed based on this interaction relationship is as follows. The degree distribution of nodes in the bipartite graph is: User degree is... Item degree is This structure reflects the sparsity of interactions in real-world scenarios.

[0180] ;

[0181] Step 2: Wavelet fusion denoising. Frequency domain processing is performed on the two-modal features after linear mapping and dimension alignment. First, they are decomposed into low-frequency approximation coefficients and high-frequency detail coefficients; then, a soft thresholding function is applied to the high-frequency detail coefficients for noise suppression; finally, fusion weights are calculated based on the consistency of cross-modal features at low and high frequencies, and weighted reconstruction is performed.

[0182] Low-frequency fusion weights This weight vector is calculated per item; the closer its value is to 0.5, the more consistent the semantic representation of the visual and textual modalities is in the low-frequency portion. In this embodiment, the weight of the item... The value of 0.502 indicates that the two modalities have the highest alignment in the low-frequency core semantics of the item.

[0183] ;

[0184] Denoising modal representation: The following shows the denoised visual features of 5 items across 8 semantic dimensions. and text features The magnitudes of the values ​​in the matrix are generally small, indicating that high-frequency noise has been effectively compressed to a neighborhood close to zero, while retaining a stable low-frequency structure that represents the core semantics.

[0185] ;

[0186] ;

[0187] Step 3: Degree-Sensitive Edge Pruning. To suppress the noise amplification effect that high-heat nodes (Hubs) in the graph may cause, a degree-sensitive edge sampling strategy is adopted. The sampling weight of the edge is set to... Prioritize preserving connections between low-degree nodes. (Pruned interaction matrix) and the symmetric normalized adjacency matrix The following formula:

[0188] ;

[0189] ;

[0190] in, The numerical value originates from symmetric normalization. The calculation shows that, in the pruned graph, the product of the degrees of the nodes at the two ends of a corresponding edge is 2. This normalization operation ensures that information is not abnormally amplified or reduced due to uneven degree distribution among nodes during subsequent message propagation.

[0191] Step 4: Learning convolution with dual-view graphs.

[0192] Heterogeneous view: The pruned and normalized bipartite graph obtained in step 3 Multi-layer graph convolution is performed to propagate messages and aggregate the outputs of each layer, thereby obtaining a user and item collaborative representation that reflects high-order collaborative relationships.

[0193] Isomorphic view: based on denoised visual modal features Based on this, construct a similarity graph between items. The graph is sparsified using the K-Nearest Neighbors (KNN) algorithm, retaining only the strongest similarity connections, thereby effectively capturing the core semantic relationships between items while suppressing weak correlation noise.

[0194] ;

[0195] As shown in the similarity map matrix, items With items The similarity between the items is 0.551. With items The similarity is 0.118, indicating that these item pairs have a stronger correlation in the visual dimension. The KNN sparsification strategy preserves such strong nearest neighbor relationships and avoids the noise accumulation and high computational cost introduced by constructing a fully connected dense graph.

[0196] Step 5: Frequency-Aware Fusion Guided by Behavior. This step is used to achieve personalized multimodal feature fusion. First, the collaborative representation is gated and modulated using the frequency-domain fusion features obtained in Step 2. Then, personalized fusion weights are generated for each item through a dual-path mechanism that includes "content-guided weights" and "cross-modal attention". The closer the weight value is to 1, the more the model focuses on the visual modality; the closer it is to 0, the more it focuses on the textual modality.

[0197] ;

[0198] The weight values ​​in this embodiment The overall distribution is close to the equilibrium value of 0.5, but slight differences in preference still exist for different items. This distribution trend is consistent with the low-frequency fusion weights calculated in step 2. The consistency corresponds to when the two modalities are more consistent in low-frequency core semantics (e.g., The fusion weights also tend to be balanced.

[0199] Step 6: Perform item recommendation prediction and model optimization. The final user and item representations are multiplied by an inner product to obtain a prediction score, which is then used to sort the data and generate a recommendation list. The following example demonstrates the process of predicting item recommendations for users. The generated Top-5 recommended item set and its corresponding predicted score vector. The closer the score is to 0, the lower the predicted preference level.

[0200] ;

[0201] ;

[0202] The output results show that, after frequency domain denoising and graph structure cleansing, the recommendation ranking is mainly driven by stable and reliable signals. Meanwhile, the personalized fusion weights generated by the BFF module, through subtle adjustments to different items, can reasonably influence the final recommendation list ranking.

[0203] In summary, the method of this application embodiment fully demonstrates the end-to-end technical process from data modeling, frequency domain denoising, graph structure optimization, dual-view learning to personalized fusion and ranking output. The effectiveness of the method of this application embodiment in frequency domain denoising, structure purification, and personalized fusion is verified through the quantifiable and verifiable intermediate matrices and vectors provided in each step.

[0204] Furthermore, embodiments of this application also provide a personalized recommendation system based on multimodal feature denoising and adaptive fusion, the system comprising:

[0205] The first module is used to construct multimodal raw features and user-item interaction graphs. The multimodal raw features include the raw features of items in different modalities. The different modalities include visual modalities or text modalities.

[0206] The second module is used to perform wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and to fuse the denoised multimodal features to obtain frequency fusion features.

[0207] The third module is used to trim the user-item interaction graph based on node degree sensitivity to obtain a normalized adjacency matrix.

[0208] The fourth module is used to construct a collaborative representation of users and items based on the normalized adjacency matrix;

[0209] The fifth module is used to construct modal representations of users and items based on the denoised multimodal features;

[0210] The sixth module is used to generate a final fused representation of user behavior patterns and modal features based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items;

[0211] The seventh module is used to perform item recommendation prediction based on the final fused representation.

[0212] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0213] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0214] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0215] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0216] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0217] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0219] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0220] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0222] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0224] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A personalized recommendation method based on multimodal feature denoising and adaptive fusion, characterized in that, The method includes the following steps: Construct multimodal raw features and user-item interaction graphs, wherein the multimodal raw features include the raw features of items in different modalities; the different modalities include visual modalities or text modalities; Wavelet transform is performed on the original multimodal features to obtain denoised multimodal features, and the denoised multimodal features are fused to obtain frequency fusion features; The user-item interaction graph is cropped based on node degree sensitivity to obtain a normalized adjacency matrix. Construct a collaborative representation of users and items based on the normalized adjacency matrix; Based on the denoised multimodal features, construct modal representations of users and items; The final fused representation of user behavior patterns and modal features is generated based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items. Item recommendation prediction is performed based on the final fused representation; The step of performing wavelet transform frequency domain denoising on the original multimodal features to obtain denoised multimodal features, and fusing the denoised multimodal features to obtain frequency fusion features, includes: The original multimodal features are spatially projected to obtain multimodal projected features; Discrete wavelet transform is performed on the multimodal projection features to obtain the low-frequency and high-frequency components of different modes; Calculate cross-modal frequency-aware similarity weights based on the low-frequency components and the high-frequency components; The low-frequency component and the high-frequency component are fused in the frequency domain according to the frequency-aware similarity weight to obtain the frequency fusion feature; The formula for calculating the frequency fusion feature is as follows: ； In the formula, Indicates frequency fusion characteristics; Represents the discrete wavelet transform; The frequency-perceived similarity weights for low-frequency components; Frequency-perceived similarity weights representing high-frequency components; Represents the low-frequency approximation coefficients in the visual modality; Represents the low-frequency approximation coefficients in the text modality; This represents the high-frequency detail coefficients after denoising of the visual modality; Represents the high-frequency detail coefficients after text modality denoising; Indicates the execution level corresponding to the discrete wavelet transform; The basis functions represent the discrete wavelet transform.

2. The method according to claim 1, characterized in that, The formula for calculating the frequency-aware similarity weight is as follows: ； In the formula, This represents the frequency-aware similarity weight; , , Indicates low-frequency components; Indicates high-frequency components; This represents a non-linear activation function.

3. The method according to claim 1, characterized in that, The step of cropping the user-item interaction graph based on node degree sensitivity to obtain a normalized adjacency matrix includes: Obtain the degree of each node in the user-item interaction graph; The sampling weight is calculated based on the degree of the node; Obtain the number of edges in the user-item interaction graph and obtain the preset pruning ratio; The number of sampling edges is determined based on the number of edges and the preset pruning ratio; The user-item interaction graph is pruned according to the normalized sampling weights to obtain the target edge set corresponding to the number of sampling edges; Construct a pruned adjacency matrix based on the target edge set; The pruned adjacency matrix is ​​normalized to obtain the normalized adjacency matrix.

4. The method according to claim 3, characterized in that, The expression for the normalized adjacency matrix is ​​as follows: ； In the formula, This represents the normalized adjacency matrix; Represents the adjacency matrix after pruning; Represents the adjacency matrix after pruning The degree matrix.

5. The method according to claim 1, characterized in that, The step of constructing a collaborative representation of users and items based on the normalized adjacency matrix includes: Initialize the ID embedding matrix; Feature propagation rules are constructed based on the normalized adjacency matrix; Based on the initialized ID embedding matrix and the feature propagation rule, the features of each layer in the user-item interaction graph are aggregated to obtain the collaborative representation of the user and items.

6. The method according to claim 1, characterized in that, The step of constructing modal representations of users and items based on the denoised multimodal features includes: Calculate the cosine similarity between items in each modality; Construct a sparse post-similarity matrix based on the cosine similarity between the items; The sparsed similarity matrix is ​​normalized to obtain the normalized similarity matrix. The item modal representation is constructed based on the normalized similarity matrix and the denoised multimodal features; Construct a user modality representation based on the item modality representation; The modal representations of the item and the user are concatenated to obtain the modal representations of the user and the item.

7. The method according to claim 1, characterized in that, The step of generating the final fused representation of user behavior patterns and modal features based on the frequency fusion features, the collaborative representation of users and items, and the modal representation of users and items includes: The frequency fusion features are injected into the collaborative representation of the user and the item to obtain the injected collaborative representation; Calculate cross-modal features based on the injected collaborative representation and the modal representations of users and items; Adaptive weights are calculated based on the injected collaborative representation, the modal representations of the user and the item, and dynamic parameters, wherein the dynamic parameters are adjusted based on the reliability of the collaborative signal; The final fused representation of user behavior patterns and modal features is calculated based on the adaptive weights, the modal representations of users and items, and the collaborative representations of users and items.

8. The method according to claim 1, characterized in that, The step of predicting item recommendations based on the final fused representation includes: The final user representation and the final item representation are calculated based on the collaborative representation of the user and the item and the final fused representation, respectively. The recommendation prediction score is determined based on the inner product of the user's final representation and the item's final representation; Item recommendations are made based on the predicted recommendation scores.

Images