Multi-modal recommendation method based on multi-scale frequency denoising

Abstract

The invention belongs to the technical field of deep learning, and provides a multi-modal recommendation method based on multi-scale frequency denoising, and the method comprises the steps: obtaining a user-article interaction data set and the multi-modal data of an article, and carrying out the embedding processing of the multi-modal data of the article, and obtaining the single-modal embedding of the article; using the user-article interaction data set and article single-mode embedding to train a multi-mode recommendation model; the multi-modal recommendation model comprises a multi-scale frequency domain denoising module, a modal specific graph learning module, a same-frequency enhanced user-article graph learning module, a dynamic modal fusion module and a preference scoring module; and outputting the preference score of each user on each article, which is obtained by the preference score module when the training is completed. According to the invention, the recommendation accuracy of the multi-modal recommendation model can be improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a multimodal recommendation method based on multi-scale frequency denoising. Background Technology

[0002] In today's digital environment, recommender systems are widely used in various online services, becoming an important tool to help users discover content of interest amidst information overload. Since user preferences are often influenced by multimodal content (such as item images and descriptive text), many researchers have turned to multimodal recommender systems, leveraging the rich auxiliary information from item multimodalities to alleviate data sparsity and improve recommendation accuracy. A typical technical process for multimodal recommender systems involves using pre-trained deep neural networks to extract features from different modalities, and then integrating these features with interaction data to enrich the representations of items and users. For example, Visual Bayesian Personalized Ranking (VBPR) incorporates visual features extracted by a pre-trained convolutional neural network (CNN) into matrix factorization within the BPR framework (Bayesian Personalized Ranking framework), achieving for the first time the effective utilization of visual content in recommender systems.

[0003] While fusing multimodal information has become a mainstream method for mitigating data sparsity and improving recommendation accuracy in recent years, most existing multimodal recommendation systems achieve feature fusion through simple operators such as concatenation, summation, or attention mechanisms, resulting in a lack of mechanisms for handling multimodal feature noise. Specifically, when processing multimodal features of items (such as images and text), the original extracted features are usually directly fused and utilized without effectively identifying and filtering out "high-frequency noise" irrelevant to the core semantics. For example, complex background textures and irrelevant pixel-level details in images, or redundant descriptive words in text, can interfere with the model's understanding of the essential semantics of items, leading to impure learned representations, hindering the learning of robust representations, and ultimately affecting recommendation accuracy. Summary of the Invention

[0004] This application aims to at least address the technical problems existing in the prior art and propose a multimodal recommendation method based on multi-scale frequency denoising.

[0005] This application provides a multimodal recommendation method based on multi-scale frequency denoising. The method includes: acquiring a user-item interaction dataset and multimodal data of items, wherein the multimodal data of items includes images and descriptive text of items; performing embedding processing on the multimodal data of items to obtain single-modal embeddings of items; and training a multimodal recommendation model using the user-item interaction dataset and the single-modal embeddings of items. The multimodal recommendation model includes: a multi-scale frequency domain denoising module, which performs frequency domain denoising and fusion on the single-modal embeddings of items based on learnable filters to obtain single-modal denoising features and multimodal fusion denoising features of items; and a modality-specific graph learning module, which learns the user-item interaction dataset, the single-modal denoising features of items, and the multimodal fusion denoising features. The system employs several modules: a noise feature module to obtain unimodal and cross-modal user-item augmentation features; a co-frequency augmentation user-item graph learning module to construct a co-frequency augmentation graph based on the user-item interaction dataset, and a graph convolutional neural network to process the graph to obtain aggregated representations of nodes, including both users and items; a dynamic modality fusion module to obtain multimodal representations of users and items based on unimodal and cross-modal user-item augmentation features and aggregated node representations; a preference scoring module to obtain each user's preference score for each item based on aggregated node representations and multimodal user-item representations; and finally, an output module to show the preference scores for each user for each item obtained by the preference scoring module upon completion of training.

[0006] The beneficial technical effects of this application are as follows: In the multimodal recommendation model, the multi-scale frequency domain denoising module transforms the single-modal embeddings of all modalities of an object into the frequency domain. This fully utilizes the characteristics of complex background textures, irrelevant pixel-level details in images, and redundant descriptive words in text, all of which are "high-frequency," effectively removing high-frequency noise in the frequency domain while retaining low-frequency information carrying core semantics. This improves information quality from the feature source and enhances the recommendation accuracy of the multimodal recommendation model. Furthermore, the multimodal recommendation model improves the discriminative power between features through a modality-specific graph learning module, effectively capturing semantic association information. The same-frequency enhanced user-item graph learning module solves the graph sparsity problem while reducing noise. The dynamic modality fusion module adaptively integrates the purified multimodal signals, thereby further improving the recommendation accuracy of the multimodal recommendation model. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a preferred embodiment of the multimodal recommendation method based on multi-scale frequency denoising according to the present invention. Figure 2 This is a schematic diagram of a multimodal recommendation model structure in a preferred embodiment of the present invention; Figure 3This is a schematic diagram of the multimodal recommendation model processing procedure in an example of the present invention. Detailed Implementation

[0008] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0009] This invention provides a multimodal recommendation method based on multi-scale frequency denoising. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, or a computer. In other words, the multimodal recommendation method based on multi-scale frequency denoising can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0010] In a preferred embodiment, the multimodal recommendation method based on multi-scale frequency denoising provided by the present invention is described in [reference needed]. Figure 1 The method includes: Step S1: Obtain the user-item interaction dataset and the multimodal data of the items, wherein the multimodal data of the items includes the images and descriptive text of the items.

[0011] It is understood that both the user-item interaction dataset and the multimodal data of the items are legally sourced and authorized by the users. Items are not limited to physical objects such as household goods, toys, clothing, and home appliances. Item images are pictures containing the item, such as toy images, images of models wearing clothing, or images of home appliances installed. Descriptive text refers to the item's title, description, category, and brand information. The multimodal data of the items includes multimodal data for all items involved in the user-item interaction dataset.

[0012] User-item interaction datasets and multimodal data of items can be pre-stored in a database, such as an e-commerce platform database. The execution entity of this method reads user-item interaction datasets and multimodal data of items from the database.

[0013] Step S2 involves embedding the multimodal data of the item separately to obtain the item's unimodal embedding. The item's unimodal embedding includes the item's visual embedding and the item's text embedding.

[0014] For example, the multimodal data of an item includes both images and descriptive text. It is not limited to processing a single item image using existing pre-trained convolutional neural networks (CNNs) or residual networks (ResNets) to obtain a single visual embedding; all item visual embeddings are arranged in a predetermined order to form an item visual embedding. Similarly, it is not limited to using a BERT-based sentence embedding model (Sentence-BERT) or a pre-trained BERT model to process a single descriptive text to obtain a single text embedding; all item text embeddings are arranged in the same predetermined order to form an item text embedding. This facilitates batch processing and improves operational efficiency. The visual and text embeddings of the same item correspond in their positions within the item visual and item text embeddings.

[0015] Step S3: Train a multimodal recommendation model using the user-item interaction dataset and item unimodal embeddings.

[0016] It is understandable that the multimodal recommendation model is trained iteratively using user-item interaction datasets and item unimodal embeddings until the training stopping condition is met. The training stopping condition is not limited to the convergence of the loss function value during the training process or the number of iterations reaching the preset maximum number of training iterations.

[0017] Step S4: Output the preference score of each user for each item obtained by the preference score module of the multimodal recommendation model when training is complete.

[0018] It is understandable that the preference score output for each user on each item will be obtained by the preference score module during the last training process when the training stopping condition is met.

[0019] In this embodiment, please see Figure 2 This demonstrates the general structure of a multimodal recommendation model. Specifically, the multimodal recommendation model includes: The multi-scale frequency domain denoising module performs frequency domain denoising and fusion on the single-mode embedding of the object based on a learnable filter, thereby obtaining the single-mode denoising features and multi-mode fused denoising features of the object. The modality-specific graph learning module obtains unimodal user and item augmentation features and cross-modal user and item augmentation features based on user-item interaction datasets, unimodal denoising features of items, and multimodal fusion denoising features. The same-frequency augmentation user-item graph learning module constructs a same-frequency augmentation graph of user-item based on the user-item interaction dataset. It then processes the same-frequency augmentation graph of user-item through graph convolutional neural network to obtain the aggregated representation of nodes, which include users and items. The dynamic modality fusion module obtains multimodal representations of users and items based on single-modal user and item augmentation features, cross-modal user and item augmentation features, and node aggregation representations. The preference scoring module obtains each user's preference score for each item based on the node's aggregated representation and the multimodal representation of users and items.

[0020] The working principles of each module in the multimodal recommendation model are explained in detail below.

[0021] In a preferred embodiment, please see Figure 3 As shown, the scale-based frequency domain denoising module includes: (1) Fast Fourier Transform Unit: The single-mode embedding of the item is transformed into the frequency domain to obtain the single-mode spectrum representation of the item.

[0022] For example, the multimodal data of an item includes images of the item. and description text The single-mode spectral representation of the item in each mode is obtained by using Fast Fourier Transform (FFT). The transformation formula is as follows: in, Represents mode, , Representing modes The complex numerical spectrum representation at frequency index k; Index in unimodal embedding of items (visual embedding of items, text embedding of items) The original feature value at that location, It is the total length of the single-modal embedding sequence of the item; Indicates rounding down; The imaginary unit, complex exponent These constitute the Fourier basis functions for frequency decomposition. This transformation reveals the spectral characteristics of each mode, enabling targeted processing for different frequency bands. Single-modal spectral characterization includes visual spectral characterization. and text spectrum representation Two types.

[0023] (2) Cross-modal fusion unit: In the frequency domain, the single-modal spectral representations of all modes of an item are fused to obtain the multimodal spectral representation of the item. Specifically, the single-modal spectral representations of all modes of an item are fused by element-wise multiplication.

[0024] For example, visual spectral representations are fused according to the following formula. and text spectrum representation Obtaining multimodal spectral characterization : ,in, This represents element-wise multiplication. The above formula, when operated in the frequency domain, corresponds to circular convolution in the spatial domain, which effectively captures inter-modal correlations while simultaneously suppressing noise interference.

[0025] (3) Filtering unit: Each single-mode spectrum representation and multi-mode spectrum representation of the item are processed by the corresponding learnable filter to obtain the single-mode filtered spectrum features and multi-mode filtered spectrum features of the item; the single-mode filtered spectrum features of the item are separated into the single-mode high-frequency spectrum features and the single-mode low-frequency spectrum features of the item.

[0026] It is understandable that a corresponding dynamic filter is set for each modality, and a corresponding dynamic filter is set for cross-modality operation. The filter is essentially a learnable weight matrix in the frequency domain, equivalent to a spatial domain convolution, and its weight matrix needs to be learned during the training of the multimodal recommendation model. Therefore, preferably, the filtering unit performs processing of each single-modal spectral representation and the multimodal spectral representation of the item using the corresponding learnable filter, including: By using learnable filters corresponding to each mode to process the single-mode spectral representation of the item in that mode, the single-mode filtered spectral features of the item in that mode are obtained. By utilizing learnable filters that correspond to cross-modal modes to process the multimodal spectral representation of an object, the multimodal filtered spectral features of the object can be obtained.

[0027] For single-mode The corresponding filter weight matrix is The processing procedure is as follows: ; This represents the spectral characteristics of a single-mode filter. When... hour, , Represents the spectral characteristics of visual filtering. This represents the visual corresponding filter. When... hour, Represents the spectral characteristics of visual filtering. This indicates the filter corresponding to the text.

[0028] For cross-modal applications, the corresponding filter weight matrix is: The processing procedure is as follows: ; This represents the spectral characteristics of multimodal filtering.

[0029] As can be seen, the aforementioned filter is a dynamic filter designed for modality-specific features and fused features, acting as a frequency-selective mask. It amplifies semantically relevant components while attenuating frequencies prone to noise. During the training of the multimodal recommendation model, this application can adaptively assign learnable weights to the frequency domain information of each modality based on the actual situation of the user-item interaction dataset and the multimodal data of the items, achieving adaptive scaling of frequency domain features, thereby solving the challenges of noise suppression and cross-modal dynamic denoising.

[0030] Leveraging the ability of frequency domain decomposition to effectively isolate fundamental semantics from detailed information across various domains, the aforementioned filter can decompose single-mode and multi-mode spectral representations into high-frequency and low-frequency components. Therefore, in a preferred embodiment of this implementation, the filtering unit performs the separation of the single-mode filtered spectral features of the item into single-mode high-frequency spectral features and single-mode low-frequency spectral features of the item, including: The frequency point sequence is obtained by sorting the values ​​of all frequency points in the single-mode filtered spectral characteristics of the item from smallest to largest. The values ​​of the first c frequency points are extracted from the frequency point sequence to form the single-mode low-frequency spectral features of the item, and the values ​​of the remaining frequency points in the frequency point sequence are used to form the single-mode high-frequency spectral features of the item; c represents the high-low frequency separation threshold, which is a positive integer and is a hyperparameter.

[0031] The above high- and low-frequency separation process can be represented as: ; ; Represents the single modality of an item The single-mode low-frequency spectral characteristics, Represents the single modality of an item The single-modal high-frequency spectral characteristics. For example, for vision... Visual low-frequency spectral characteristics Visual high-frequency spectrum characteristics For text Low-frequency spectral features of text High-frequency spectral features of text .

[0032] It can be seen that the spectral characteristics of single-mode filtering The smallest c frequency points (i.e., elements) are classified as single-modal low-frequency spectral features, while the remaining elements are classified as single-modal high-frequency spectral features. Low-frequency components are typically smoother and contain thematic content information of the modality (e.g., the main object in an image or entity nouns in text). In contrast, high-frequency components typically exhibit rapid changes and are prone to carrying noise (e.g., edges and background in an image, or emotionally charged modifiers in text).

[0033] (4) Inverse Fast Fourier Transform unit, which converts the single-mode low-frequency spectrum features, single-mode high-frequency spectrum features and multi-mode filtered spectrum features of the item to the spatial domain, respectively, to obtain the single-mode low-frequency features, single-mode high-frequency features and multi-mode fused denoising features of the item.

[0034] Specifically, the formula for the Inverse Fast Fourier Transform (IFFT) can be expressed as: in, Indicates the modality of the item The single-mode low-frequency characteristics in the index Spatial threshold at that location, Indicates the modality of the item The single-mode low-frequency characteristics; Indicates the modality of the item The single-mode high-frequency characteristics in the index Spatial threshold at that location Indicates the modality of the item The single-mode high-frequency characteristics; Multimodal fusion denoising features representing items in the index Spatial threshold at that location This represents the multimodal fusion denoising features of an item.

[0035] For example, items in vision The visual low-frequency features are represented as , objects in vision The visual high-frequency features are represented as ; Items in text The low-frequency features of the text are represented as Items in text The high-frequency features of the text are represented as .

[0036] (5) Weighting unit: The single-mode low-frequency features and single-mode high-frequency features of the item in each mode are weighted to obtain the single-mode denoising features of the item in that mode. The high-frequency part that is prone to noise is weighted to suppress noise.

[0037] It is understandable that by setting a high-frequency suppression weight for each mode, the mode... The high-frequency suppression weight is , which can be a hyperparameter. The following weighted formula is used to obtain the item's modal... Single-mode denoising features (use (Representing items): in, Indicates the modality of the item The single-mode low-frequency characteristics, Indicates the modality of the item The single-mode high-frequency characteristics.

[0038] For example, items in vision The visual denoising features in the modality are represented as follows Items in text The text denoising features in the modal are represented as follows To differentiate from users, subsequent use Represents the multimodal fusion denoising features of an item. .

[0039] It is evident that the weighted unit effectively reduces noise while preserving semantically important information, and ensures the smooth decay of high-frequency components while maintaining the differentiability required for gradient optimization.

[0040] In a preferred embodiment, please refer to Figure 2 and Figure 3 The modality-specific graph learning module executes: Step A1: Construct a fully connected unimodal item graph for all items in the user-item interaction dataset in that modality based on the unimodal denoising features of the items in each modality.

[0041] Understandably, each modality constructs a unimodal item fully connected graph, such as a visual fully connected graph and a text fully connected graph. In the unimodal item fully connected graph, the nodes are all items in the user-item interaction dataset, and an edge is set between any two nodes. The weight of each edge is the similarity between the unimodal denoised features of the two items connected by the edge. Specifically, the cosine similarity formula can be used.

[0042] For example, from the item in modality Single-mode denoising features Extract the single-modal denoising features of item a and item b. and Calculate the modalities of items a and b. similarity :

[0043] Modality The fully connected graph of a single-modal item can be represented as follows: .

[0044] Step A2: Prune and / or normalize the fully connected graph of unimodal items for each modality to obtain the unimodal item association matrix for that modality.

[0045] It is understandable that in step A2, only pruning or normalization can be performed, or pruning can be performed first and then normalization.

[0046] By pruning to preserve the most relevant connections and ensure computational efficiency, specifically, the following formula is used to retain only the modal connections for each item. The former Pruning the edge of the graph:

[0047] It is a positive integer, and it is a modal. The number of edges to retain. Represents a collection of items. Indicates that items a and d are in modal Similarity, i.e., edge The weight. Representing modes The edge with the largest weight in the fully connected graph of the unimodal item is... The set of edge weights consists of the weights of each edge. The formula above indicates that when the edge... weight When the edge belongs to the above set of edge weights, retain the edge. When the side weight If the edge does not belong to the above set of edge weights, delete the edge. . After pruning, it can be represented as .

[0048] Normalizing the fully connected graph of unimodal items, or a pruned fully connected graph of unimodal items, can alleviate potential gradient instability issues during training. For example, normalizing the modality... Pruned single-modal item fully connected graph The normalization formula is: in, yes The degree matrix, Representing modes A fully connected graph of single-modal items.

[0049] Step A3, please see Figure 3 The max pooling strategy is used to integrate the unimodal item association matrices of all modalities to obtain the fused item association matrix.

[0050] Understandably, the max-pooling strategy is used to integrate cross-modal complementary information, and a fused item association matrix is ​​constructed by selecting the maximum edge weight of each item pair among all modal pairs. Merge item association matrix Weights of the edges between items a and b for: in, Representing modes Normalized unimodal item fully connected graph Middle The weight, Modal Normalized unimodal item fully connected graph Middle The weight.

[0051] Step A4: Obtain the single-modal item enhancement features of the item based on the single-modal item association matrix and the single-modal denoising features of the item.

[0052] It is understandable that the item is in modality Single-modal item enhancement features The calculation formula is: Representing modes A fully connected graph of single-modal items. Indicates the modality of the item Single-modal denoising features. For example, visual... In modality, objects in vision Visual item enhancement matrix ;text In modality, items are in the text Text Item Enhancement Matrix .

[0053] Step A5: Obtain cross-modal item enhancement features based on the fused item association matrix and the multimodal fused denoising features of the items.

[0054] It is understandable that cross-modal item enhancement features... The calculation formula is: Represents the multimodal fusion denoising features of an item. This represents the association matrix of merged items.

[0055] It should be noted that steps A4 and A5 can be executed in parallel or sequentially, and there is no limitation on this.

[0056] Step A6: Construct a user-item bipartite graph based on the user-item interaction dataset. Aggregate the sub-features of each user's neighboring items in the unimodal item enhancement features in the user-item bipartite graph to obtain the unimodal features of each user. Aggregate the sub-features of each user's neighboring items in the cross-modal item enhancement features in the user-item bipartite graph to obtain the cross-modal features of each user.

[0057] It's understandable that nodes in a user-item bipartite graph include two types: user nodes and item nodes. There is an edge between user nodes and item nodes that interact, and no edge between user nodes and item nodes that do not interact. There are no edges between user nodes and no edges between item nodes. In a user-item bipartite graph, the neighboring items of each user node are those that have interacted with that user, i.e., item nodes that have a connecting edge to that user node.

[0058] Understandably, users In modality Single-modal characteristics The calculation formula is: user The formula for calculating cross-modal characteristics is: in, Represents users in a user-item bipartite graph. A collection of neighboring items, express Quantity of items in the middle neighborhood; express Middle Neighborhood Items The set of neighboring users, that is, the set of neighboring items The set of users with connected edges. express Number of users in the middle neighborhood Indicates neighboring items In modality Single-modal item enhancement features Sub-features in. Indicates neighboring items Denoising features in multimodal fusion Sub-features in. Normalization factor. Ensure that information is aggregated in a balanced manner from neighborhoods of different sizes. Single-layer propagation is sufficient to capture essential semantic relationships, as deeper architectures may introduce additional noise and degrade performance. Normalization is introduced to maintain numerical stability.

[0059] Step A7: Concatenate the unimodal item augmentation features of each modality with the unimodal features of all users to obtain the unimodal user and item augmentation features for that modality. By concatenating cross-modal item augmentation features and cross-modal features of all users, cross-modal user and item augmentation features can be obtained. . , Represents a set of users The number of users, Represents a set of items The number of items, express , , , The feature dimensions.

[0060] In this embodiment, the modality-specific graph learning module improves feature discrimination and extracts high-order representations by introducing modality-specific K-nearest neighbor graphs, and effectively captures semantic association signals through structured item-item relationships.

[0061] Traditional recommender system methods often employ pruning strategies on the user-item bipartite graph to eliminate noisy interactions. While these methods are effective in denoising, they inevitably exacerbate the inherent sparsity of the user-item bipartite graph structure, potentially discarding valuable behavioral patterns and weakening the model's ability to capture complex user preferences. Inspired by frequency domain signal processing techniques, this application proposes a novel co-frequency enhanced graph construction paradigm (i.e., a co-frequency enhanced user-item graph learning module) that simultaneously addresses the dual challenges of noise reduction and graph sparsity. This relies on the fact that meaningful user-item interactions exhibit characteristic frequency patterns distinguishable from random noise. Unlike traditional co-occurrence-based methods, the frequency enhancement method of the co-frequency enhanced user-item graph learning module in this application can capture both direct interactions and implicit behavioral associations. Therefore, in a preferred embodiment, the co-frequency enhanced user-item graph learning module includes: (1) Graph augmentation unit, the graph augmentation unit executes: Step B1: Obtain the user-item interaction matrix based on the user-item interaction dataset.

[0062] Understandably, in the user-item interaction matrix, each element represents whether a user interacts with a particular item. If there is interaction, the element is 1; otherwise, it is 0. This represents the user-item interaction matrix.

[0063] Step B2: Transform the user-item interaction matrix to obtain the user-user frequency matrix and the item-item frequency matrix.

[0064] It is understandable that there is a frequency matrix between items. User-User Frequency Matrix Item-Item Frequency Matrix Each element in the table represents the frequency of interaction between two items, such as... Item-item frequency matrix Chinese items With items Interaction frequency. User-user frequency matrix. Each element in the table represents the frequency of interaction between two users, such as... Represents the user-user frequency matrix Chinese users With users The frequency of interaction.

[0065] Step B3: Process the user-user frequency matrix and the item-item frequency matrix using a frequency-aware filtering method to obtain the user-user frequency enhancement matrix and the item-item frequency enhancement matrix, respectively. More preferably, the frequency-aware filtering method updates the value of each element in the user-user frequency matrix or the item-item frequency matrix as follows: Iterate through each element in the user-user frequency matrix or item-item frequency matrix. If the value of the currently iterated element belongs to the top of the user-user frequency matrix or item-item frequency matrix... If the value of the currently traversed element is greater than the preset frequency threshold, then the value of the currently traversed element is updated to the product of the value of the currently traversed element and the frequency enhancement coefficient; otherwise, the value of the currently traversed element remains unchanged. It is a positive integer.

[0066] For example, for a user-user frequency matrix The formula for the frequency-aware filtering method is: in, This represents the user-to-user frequency enhancement matrix. express Chinese users With users The frequency of interaction; This represents the user-to-user frequency enhancement factor. The user-to-user frequency threshold can be a hyperparameter. express belong The largest value One of the elements.

[0067] For example, an item-item frequency matrix The formula for the frequency-aware filtering method is: in, This represents the item-item frequency enhancement matrix. express Chinese items With items The frequency of interaction; This represents the item-item frequency amplification factor. This represents the item-item frequency threshold. express belong The largest value One of the elements.

[0068] The frequency-aware filtering method described above aligns the user-user frequency enhancement matrix and the item-item frequency enhancement matrix with the user-item interaction matrix and extracts principal components to suppress noise effects. , It can preserve meaningful connections while filtering sparse and potentially noisy interactions. Through , Weights are assigned to each retained frequency domain enhancement element so that it can learn effectively without over-amplifying and distorting the information in the original interaction matrix.

[0069] Step B4: Construct a user-item frequency enhancement graph using the user-user frequency enhancement matrix, the item-item frequency enhancement matrix, and the user-item interaction matrix. Specifically, this step includes: Step B41: Construct the initial user-item frequency enhancement graph according to the following formula. : .

[0070] Step B42: Construct the user-item resonance enhancement graph according to the following formula. Symmetric normalization is used to ensure numerical stability: in, Enhanced graph for initial user-item synchronization The degree matrix.

[0071] (2) Node embedding module, which propagates node ID embedding in the user-item frequency enhancement graph. Node ID embedding includes user ID embedding and item ID embedding.

[0072] In this embodiment, the node ID embedding is a learnable parameter that is continuously optimized during the training of the multimodal recommendation model. During the initial training, the node ID embedding can be randomly initialized. The user-item frequency enhancement graph includes user nodes and item nodes; therefore, there are two types of node ID embeddings: user ID embedding and item ID embedding.

[0073] (3) Graph convolutional neural network: perform graph convolution operation on the user-item frequency enhancement graph after embedding the propagation node ID to obtain the aggregate representation of the node.

[0074] Graph convolutional neural networks include Layered graph convolution, It is a positive integer. Indicates the index of the graph convolutional layer number. . No. The message passing operation of layer graph convolution is defined as follows: in, Indicates the first Nodes in layer graph convolution User-item synchronization enhancement feature (possibly user node or item node), For all nodes The initial embedding, which is continuously optimized during the training of the multimodal recommendation model, is learnable. To comprehensively integrate multi-scale neighborhood information, a layer aggregation mechanism is used to merge the representations of all layers: in, Represents all nodes The aggregated representation contains both high-order collaborative information and interaction frequency signals from the graph convolutional layers. Middle user node The aggregation feature is represented as , Item nodes The aggregation feature is represented as .

[0075] In this implementation, the user-item frequency-enhanced graph has three major advantages over traditional methods: First, it preserves meaningful connections that have been discarded by aggressive pruning; second, it enhances semantically important relationships through frequency-based weighting; and finally, it maintains graph connectivity while effectively suppressing noise, thereby resolving the fundamental contradiction between denoising and sparsity in recommendation graphs.

[0076] In acquiring enhanced features for cross-modal users and items and nodes After aggregating and representing the data, the key lies in effectively fusing them to achieve the final prediction. In real-world scenarios, user decisions often stem from complex interactions between unimodal and cross-modal preferences. To capture this nuance, this application designs a dynamic modality fusion module that adaptively integrates the purified multimodal signals based on user behavior.

[0077] In a preferred embodiment, please see Figure 3 The dynamic modal fusion module includes: (1) Attention mechanism unit, which processes cross-modal user and item augmentation features based on the attention mechanism. Combine unimodal user and item enhancement features across all modalities to obtain reweighted unimodal features.

[0078] It is understandable that this includes: First, attention mechanisms are used to process cross-modal user and item augmentation features respectively. This yields adaptive weights for each modality. For example, for a modality... Adaptive weights for: in, Represents the normalized exponential function, Represents the hyperbolic tangent function; Representing modes projection matrix The inverted matrix, Representing modes Attention weights Representing modes The bias. , , All of these are learnable parameters that are optimized and updated during the training of the multimodal recommendation model.

[0079] Finally, the reweighted single-modal features are obtained by processing them according to the following formula. : in, Representing modes Unimodal user and item augmentation features, This represents the set of modal types.

[0080] (2) The distillation processing unit uses the gating mechanism to obtain the single-modal explicit preference matrix and cross-modal explicit preference matrix of each modality based on the node aggregation representation.

[0081] For mode Single-modal explicit preference matrix obtained using gating mechanism for: in, Represents learnable modalities Gating weights, Represents learnable modalities The bias, Represents a node Aggregate representation, nodes For user nodes or item nodes. This represents the sigmoid function.

[0082] The cross-modal display preference matrix obtained using the gating mechanism is represented as follows: : in, Represents learnable cross-modal gating weights. This represents a learnable cross-modal bias. , , , All of these are continuously optimized during the training process of the multimodal recommendation model.

[0083] The aforementioned gating mechanism adjusts the influence of each modality based on the user's historical interactions, accurately reflecting user preferences.

[0084] (3) Coordination unit: Based on cross-modal explicit preference matrix, reweighted single-modal features, cross-modal user and item enhancement features, and single-modal explicit preference matrix of all modalities, multimodal representation of users and items is obtained.

[0085] It is understandable that the multimodal representation of users and items can be obtained using the following formula. : .

[0086] In a preferred embodiment, the preference scoring module performs: Step C1: Concatenate the aggregated representation of each user and the features corresponding to that user in the multimodal representation of users and items to obtain the final embedding of that user.

[0087] For example, users The final embedding is: in, express Chinese users The aggregation characterization, Multimodal representation of users and items Chinese users Corresponding characteristics.

[0088] Step C2: Concatenate the aggregated representation of each item and the features corresponding to that item in the multimodal representation of the user and the item to obtain the final embedding of the item.

[0089] For example, items The final embedding is:

[0090] in, express Chinese items The aggregation characterization, Multimodal representation of users and items Chinese items Corresponding features.

[0091] It should be noted that steps C1 and C2 can be executed in parallel or sequentially, and this application does not impose any restrictions on this.

[0092] Step C3: Obtain each user's preference score for each item based on each user's final embedding and each item's final embedding.

[0093] For example, users For items The preference score is: in, express The transpose of .

[0094] In a preferred embodiment, a modality alignment and separation strategy is employed during the training of the multimodal recommendation model, and the loss function during the training process of the multimodal recommendation model is... Including spectral harmonic loss Bayesian personalized ranking loss and contrastive learning loss Among them, spectral harmonic loss Including modal alignment loss and feature separation loss .

[0095] Among them, the Bayesian personalized ranking loss is: This represents the training triple set. , Represents items and items . This indicates that positive sample pairs have been observed. Indicates user Preferred items . This represents the negative sample pairs. Indicates user Dislike of items . Indicates user For items Preference scores.

[0096] , , represents the first weight, second weight, and third weight, respectively, all of which are hyperparameters, and respectively contribute to the weighted spectral harmonic loss, contrastive loss, and weight decay. This represents all learnable parameters during the training of the multimodal recommendation model. This is a regularization term.

[0097] Spectral harmonic loss ,in, in, , These represent the visual noise reduction features of the items. The mean and variance of , Text denoising features for items The mean and variance of L1 norm. Spectral harmonic loss. Alignment promotes similar distribution characteristics among different modes in the latent space, thereby enhancing their fusion compatibility. While alignment can promote cross-modal consistency, preserving the unique information within each mode is equally important.

[0098] To prevent over-alignment and preserve modality-specific features, a feature separation loss is introduced. To enhance the difference between original and transformed features: for visual features, maximize the visual embedding of the item. Visual noise reduction features of objects Dissimilarity between features. Similarly, for text features, maximize the text embeddings of all items. Text denoising features of items The degree of separation between them. This objective is achieved through a comparative divergence mechanism: in, Calculate cosine similarity. Measuring Euclidean distance, This is a balance coefficient used to control the balance between minimizing similarity and maximizing distance. This loss ensures that the denoising process removes noise while preserving necessary mode-specific information. , The combined effect enables the model in this application to learn statistically aligned and informationally discriminative representations, ultimately achieving more robust multimodal recommendation.

[0099] Comparative learning loss To ensure alignment between the behavior-centric and modality-centric perspectives, a contrastive learning objective based on InfoNCE loss is employed to enhance the robustness of learned preferences. , This represents the contrastive learning loss on the user side. This indicates the comparative loss on the item side.

[0100]

[0101] Among them, temperature parameter Controlling the sharpness of the probability distribution is crucial for preserving the core information encapsulated in behavioral and modal representations. Similarly, by replacing user entities with items, the item-side contrastive loss can be derived from the above formula. . , , , express Chinese items Aggregate representation of item j, user u, and user v , , , Multimodal representation of users and items User v, User u, Item The characteristics corresponding to item j.

[0102] In the modality alignment and separation strategy, frequency-based filtering addresses the inherent distributional differences between visual and textual modalities while preserving their unique features. A spectral harmonic loss consisting of two loss functions is introduced: modality alignment loss and feature separation loss. These two objectives work synergistically to ensure the model effectively learns shared and unique information across modalities. The modality alignment loss minimizes the distributional gap between visual and textual representations in the latent space. By aligning their statistical properties, it promotes better cross-modal integration and enhances the model's ability to capture complementary information.

[0103] Experimental validation was performed on the multimodal recommendation model (hereinafter referred to as the MSCF-Net model) provided in this application: 1. Experimental Dataset Setup This experiment uses three commonly used categories from the Amazon review dataset: Baby Products, Sports & Outdoors, and Clothing, Footwear & Jewelry (referred to as Baby, Sports, and Clothing, respectively). To ensure data quality, both users and items were filtered using a 5-core filter. Specific dataset statistics are shown in Table 1 below. For item representation, we use publicly available feature vectors: including 4,096-dimensional visual features extracted through a pre-trained CNN, and 384-dimensional text embeddings generated by combining product titles, descriptions, categories, and brand information using Sentence-BERT.

[0104] Table 1. Dataset Statistics

[0105] 2. Baseline Model To verify the effectiveness of the MSCF-Net model, it is compared with two major categories of recommendation models: a general model that uses only interaction data, and a multimedia model that utilizes both interaction information and multimodal features, as detailed below: 1) MF is based on the matrix factorization framework and uses the Bayesian personalized ranking loss function to model user preferences only through user-item interaction datasets.

[0106] 2) LightGCN simplifies graph convolutional networks by removing nonlinear activation and feature transformation. Its core design includes lightweight graph convolution operations and layer embedding combination mechanisms, and is optimized for efficient collaborative filtering.

[0107] 3) VBPR incorporates visual features into the traditional BPR model, combining visual embeddings with item ID representations and placing them within a matrix factorization framework to improve recommendation performance.

[0108] 4) MMCGN constructs independent graph structures for different modalities, learns user and item representations from each modal graph, and finally predicts user-item interactions by fusing these representations.

[0109] 5) SLMRec introduces self-supervised learning into recommendation systems, mining the intrinsic relationships in multimodal item data through node self-discrimination tasks without relying on external annotations.

[0110] 6) MGCN uses a multi-view network to handle modal noise and user interaction sparsity issues. It integrates behavior-aware denoising, independent view encoders and self-supervised fusion modules, and performs well in sparse scenarios.

[0111] 7) FREEDOM proposes an efficient graph structure processing method, proving that the performance of pre-computed frozen item graphs is comparable to that of dynamically learned structures. By fixing the k-nearest neighbor graph and pruning the user-item interaction graph, it significantly reduces training costs while maintaining accuracy advantages, making it suitable for large-scale application scenarios.

[0112] 8) LGMRec learns user interests from both local and global perspectives. It decouples collaborative signals and modal signals locally, models global cross-modal associations through hypergraphs, and achieves advanced performance in sparse user scenarios by combining contrastive learning techniques.

[0113] 9) The EVEN system handles semantic and behavioral noise. Through a joint learning framework of task relevance evaluation and gradient-aware pruning, it simultaneously optimizes the item graph and user-item graph, which improves the robustness of the model and accelerates training convergence without introducing additional inference overhead.

[0114] 3. Evaluation Indicators To comprehensively evaluate the performance of the MSCF-Net model in this application, two standard evaluation metrics in recommender systems, Recall@K and NDCG@K, were used, with cutoff values ​​K set to 10 and 20. This setting allows for multi-dimensional evaluation of the recommendation list quality at different depths. To ensure statistical reliability and reduce the impact of randomness, all experiments were independently repeated 5 times, and the final results were based on the average score. Recall@K quantifies the model's coverage ability by calculating the proportion of successfully retrieved truly relevant items in the top K recommendations, reflecting its effectiveness in capturing user preferences. NDCG@K, on ​​the other hand, provides a more refined evaluation of ranking accuracy by assigning higher weights to top-ranked items, thus measuring the model's ability to prioritize the most relevant items. Higher values ​​for both metrics indicate better recommendation performance.

[0115] 4. Model performance As shown in Table 2, the performance comparison results indicate that, in comparison with various advanced recommendation models, the following conclusions can be drawn: (1) Superiority of the MSCF-Net model: The comprehensive experimental results show that the MSCF-Net model achieves significant and consistent performance improvements over the existing best baseline model on all datasets and evaluation metrics (statistical significance verification p≤0.05). We attribute its superiority to the model's ability to use frequency domain characteristics for robust multimodal representation learning: by mining discriminative spectral patterns, it effectively captures general cross-modal semantic information while suppressing modality-specific noise using a dynamic filtering mechanism; it encodes high-order cooperative signals and semantically associated item relationships with the help of a multimodal graph learning module; and it adaptively balances single-modality and fused signals based on enhanced behavioral representations to ensure the best fit with real decision-making patterns.

[0116] (2) The key role of multimodal information: Experimental results once again verify the irreplaceable role of using rich multimodal content to alleviate data sparsity. Although baseline models such as VBPR and MMGCN have confirmed the effectiveness of fusing modal features, their performance is often limited by inherent noise. The MSCF-Net model promotes the development of this paradigm by actively purifying multimodal signals rather than simply fusing them—extracting purer and more semantically consistent representations from visual text data, proving the core value of the quality of multimodal information (rather than its simple existence) for achieving robust recommendation performance.

[0117] (3) Effectiveness of Frequency Domain Signal Processing: The significant performance gap between the MSCF-Net model and the strongest baseline EVEN particularly highlights the advantages of the frequency domain processing path in this scheme. Although EVEN effectively handles semantic and behavioral noise through graph structure evaluation and pruning, its main work is performed in the spatial domain. In contrast, the spectral filtering of the MSCF-Net model provides a more fundamental and fine-grained solution by decoupling noise (often concentrated in high-frequency components) from semantic content (mainly present in low-frequency components) before graph propagation. This pre-denoising and enhancement mechanism implemented in the frequency domain results in a cleaner initial representation and more effective subsequent graph learning, which has a significant advantage over spatial domain denoising methods.

[0118] Table 2 Performance comparison of MSCF-Net model and baseline models on three real datasets.

[0119] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0120] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal recommendation method based on multi-scale frequency denoising, characterized in that, The method includes: Acquire user-item interaction datasets and multimodal data of items, where the multimodal data of items includes images and descriptive text of items; The multimodal data of the item are embedded separately to obtain the item's unimodal embedding; A multimodal recommendation model is trained using a user-item interaction dataset and item unimodal embeddings; the multimodal recommendation model includes: The multi-scale frequency domain denoising module performs frequency domain denoising and fusion on the single-mode embedding of the object based on a learnable filter, thereby obtaining the single-mode denoising features and multi-mode fused denoising features of the object. The modality-specific graph learning module obtains unimodal user and item augmentation features and cross-modal user and item augmentation features based on user-item interaction datasets, unimodal denoising features of items, and multimodal fusion denoising features. The same-frequency augmentation user-item graph learning module constructs a same-frequency augmentation graph of user-item based on the user-item interaction dataset. It then processes the same-frequency augmentation graph of user-item through graph convolutional neural network to obtain the aggregated representation of nodes, which include users and items. The dynamic modality fusion module obtains multimodal representations of users and items based on single-modal user and item augmentation features, cross-modal user and item augmentation features, and node aggregation representations. The preference scoring module obtains each user's preference score for each item based on the node's aggregated representation and the multimodal representation of users and items. Output the preference score for each user on each item obtained by the preference score module when training is complete.

2. The method according to claim 1, characterized in that, The multi-scale frequency domain denoising module includes: The Fast Fourier Transform unit converts the single-mode embedding of an object into the frequency domain to obtain the single-mode spectral representation of the object. The cross-modal fusion unit fuses the single-modal spectral representations of all modes of an object in the frequency domain to obtain the multimodal spectral representation of the object. The filtering unit processes each single-mode spectral representation and the multi-mode spectral representation of the item using corresponding learnable filters to obtain the single-mode filtered spectral features and multi-mode filtered spectral features of the item; and separates the single-mode filtered spectral features of the item into the single-mode high-frequency spectral features and the single-mode low-frequency spectral features of the item. The inverse fast Fourier transform unit transforms the single-mode low-frequency spectral features, single-mode high-frequency spectral features, and multi-mode filtered spectral features of the object into the spatial domain, respectively, to obtain the single-mode low-frequency features, single-mode high-frequency features, and multi-mode fused denoising features of the object; The weighting unit performs weighted processing on the single-mode low-frequency features and single-mode high-frequency features of the item in each mode to obtain the single-mode denoising features of the item in that mode.

3. The method according to claim 2, characterized in that, The filtering unit performs processing of each single-modal spectral representation and the multimodal spectral representation of the item using corresponding learnable filters, including: By using learnable filters corresponding to each mode to process the single-mode spectral representation of the item in that mode, the single-mode filtered spectral features of the item in that mode are obtained. By utilizing learnable filters that correspond to cross-modal modes to process the multimodal spectral representation of an object, the multimodal filtered spectral features of the object can be obtained.

4. The method according to claim 2, characterized in that, The filtering unit performs the separation of the single-mode filtered spectral features of the item into single-mode high-frequency spectral features and single-mode low-frequency spectral features of the item, including: The values ​​of all frequency points in the single-mode filtered spectral features of the item are sorted in ascending order to obtain a frequency point sequence. The values ​​of the first c frequency points are extracted from the frequency point sequence to form the single-mode low-frequency spectral features of the item. The values ​​of the remaining frequency points in the frequency point sequence are used to form the single-mode high-frequency spectral features of the item. c is the high-low frequency separation threshold, which is a positive integer.

5. The method according to any one of claims 1-4, characterized in that, The modality-specific graph learning module performs the following: Based on the unimodal denoising features of items in each modality, construct a unimodal fully connected graph of all items in the user-item interaction dataset in that modality; For each modality, the unimodal item fully connected graph is pruned and / or normalized to obtain the unimodal item association matrix for that modality. A max-pooling strategy is used to integrate the unimodal item association matrices of all modalities to obtain a fused item association matrix; The single-modal item enhancement features of the item are obtained based on the single-modal item association matrix and the single-modal denoising features of the item; Cross-modal item enhancement features are obtained by fusing the item association matrix and the multimodal denoising features of the items; Based on the user-item interaction dataset, a user-item bipartite graph is constructed. Sub-features of the neighboring items of each user in the unimodal item augmentation features are aggregated to obtain the unimodal features of each user. Sub-features of the neighboring items of each user in the cross-modal item augmentation features are aggregated to obtain the cross-modal features of each user. By concatenating the unimodal item augmentation features of all items and the unimodal features of all users for each modality, the unimodal user and item augmentation features for that modality are obtained. By combining cross-modal item augmentation features with cross-modal features of all users, cross-modal user and item augmentation features are obtained.

6. The method according to any one of claims 1-4, characterized in that, The same-frequency enhanced user-item graph learning module includes: The graph enhancement unit obtains a user-item interaction matrix based on the user-item interaction dataset; it transforms the user-item interaction matrix to obtain a user-user frequency matrix and an item-item frequency matrix; it processes the user-user frequency matrix and the item-item frequency matrix using a frequency-aware filtering method to obtain a user-user frequency enhancement matrix and an item-item frequency enhancement matrix; and it constructs a user-item frequency enhancement graph using the user-user frequency enhancement matrix, the item-item frequency enhancement matrix, and the user-item interaction matrix. The node embedding module propagates node ID embeddings in the user-item frequency enhancement graph. Node ID embeddings include user ID embeddings and item ID embeddings. A graph convolutional neural network is used to perform graph convolution operations on the user-item frequency enhancement graph after embedding the propagation node ID to obtain the aggregated representation of the nodes.

7. The method according to claim 6, characterized in that, The frequency-aware filtering method updates the value of each element in the user-user frequency matrix or item-item frequency matrix as follows: Iterate through each element in the user-user frequency matrix or item-item frequency matrix. If the value of the currently iterated element belongs to the top of the user-user frequency matrix or item-item frequency matrix... If the value of the currently traversed element is greater than a preset frequency threshold, then the value of the currently traversed element is updated to the product of the current element's value and the frequency enhancement coefficient; otherwise, the value of the currently traversed element remains unchanged. It is a positive integer.

8. The method according to any one of claims 1-4 and 7, characterized in that, The dynamic modality fusion module includes: The attention mechanism unit processes cross-modal user and item augmentation features and all-modal unimodal user and item augmentation features based on the attention mechanism to obtain reweighted unimodal features; The distillation processing unit uses a gating mechanism to obtain the single-modal explicit preference matrix and cross-modal explicit preference matrix for each modality based on node aggregation representation; The coordination unit obtains multimodal representations of users and items based on cross-modal explicit preference matrices, reweighted unimodal features, cross-modal user and item augmentation features, and unimodal explicit preference matrices of all modalities.

9. The method according to any one of claims 1-4 and 7, characterized in that, The preference scoring module performs the following: By concatenating the aggregated representation of each user and the features corresponding to that user in the multimodal representation of users and items, the final embedding of that user is obtained; By concatenating the aggregated representation of each item with the features corresponding to that item in the multimodal representation of the user and the item, the final embedding of the item is obtained; Each user's preference score for each item is obtained based on the final embedding of each user and the final embedding of each item.

10. The method according to claim 1, characterized in that, The loss functions in the training process of multimodal recommendation models include spectral harmonic loss, Bayesian personalized ranking loss, and contrastive learning loss; among them, spectral harmonic loss includes modality alignment loss and feature separation loss.

Images