A multi-modal recommendation-oriented debiasing model construction method

Abstract

The application discloses a kind of multi-modal recommendation-oriented debiasing model construction methods, specifically including input feature and alignment, obtain the same dimension of item visual feature and item text feature;Multi-modal graph preference fusion, obtain fusion item vector and user aggregation semantic feature;Based on fusion item vector, calculate the causal bias term of multi-modal item feature;Prediction and deviation correction;Stage joint training etc.Steps.The application can effectively suppress multi-modal noise, accurately capture modal preference, remove recommendation bias from the causal level, significantly improve recommendation accuracy, fairness and generalization ability.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, recommendation systems, and multimodal learning technology, and specifically relates to a method for constructing a bias-free model for multimodal recommendation. Background Technology

[0002] The explosive growth of online content has made recommender systems a key tool for alleviating information overload and improving user engagement. Multimodal recommender systems (MRSs) integrate multiple data sources, including implicit user-item interactions, textual feedback, and visual content, to capture cross-modal insights that cannot be obtained from unimodal methods. They significantly outperform traditional methods in both recommendation accuracy and user satisfaction, and have therefore become a core focus of current research.

[0003] Early multimodal recommendation methods generally followed a relevance-fitting approach, optimizing the recommendation objective by splicing multimodal features of items with interaction signals. However, in real-world deployment scenarios, the OOD (Out-of-Demand) failure problem caused by out-of-distribution data becomes prominent, with the trained multimodal features being confounded by a large number of factors unrelated to users' true preferences. Especially in multimodal recommendation scenarios, models are prone to misattributing high-exposure but non-user-preferred visual styles and high click-through rates as core preference criteria, and treating non-causally related confounding factors such as brand and promotional attributes as core features for recommendation decisions. Ultimately, this leads to recommendation results that deviate from users' actual needs, resulting in suboptimal or even unfair recommendation problems.

[0004] To address the problems inherent in correlation-based fitting approaches, academia and industry have recently initiated a paradigm shift towards causal paradigms. The core idea is to learn causal representations of true user preferences that are invariant to confounding factors through intervention estimation. Technically, this shift is mainly divided into two categories: mature paradigms and emerging frontiers. Mature paradigms, represented by structural causal models, explicitly construct a ternary causal graph of users, items, and confounding factors on observed data, using backdoor and frontdoor adjustments to identify causal effects and optimize debiasing strategies. However, this method faces three challenges when applied to multimodal recommendation scenarios: high-dimensional features, dynamically changing associations, and multi-source heterogeneous fusion. Confounding factors are unobservable, and backdoor paths are difficult to pinpoint accurately. Emerging frontier solutions for social cross-domain migration and generative causal simulation are limited by the assumption of a single-modal isomorphic scenario. Faced with the heterogeneous and high-dimensional characteristics of visual and textual features in multimodal recommendation, they cannot accurately track the flow path of confounding factors across modal spaces, nor can they adaptively optimize debiasing strategies in dynamically changing multi-source features. Meanwhile, neither of the two approaches can effectively distinguish between the learned true causal relationships and the false statistical correlations brought about by multimodal features. This allows the backdoor path opened by the item-side confusion factor to be unobstructed, which seriously affects the generalization ability in scenarios such as long-tail items and makes it difficult to meet the actual needs of accurate bias removal in multimodal recommendation scenarios. Summary of the Invention

[0005] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing a method for constructing a bias-free model for multimodal recommendation.

[0006] The technical solution of this invention is: a method for constructing a bias-free model for multimodal recommendation, which is carried out in the following steps: Step 1. Input features and alignment Acquire visual, textual, and user interaction data of items, unify the dimensions of visual and textual features, and obtain item visual and textual features of the same dimension; Step 2. Multimodal graph preference fusion A unimodal graph is constructed based on the visual and textual features of the items. The graph is then aggregated and fused across modalities using a graph neural network constrained by a normalized sparse adjacency matrix to obtain a fused item vector. For the fused item vectors that have been interacted with by the user, the user identifier is combined with attention-weighted aggregation to output the aggregated semantic features of the user. Step 3. Calculate the causal bias term of multimodal item features based on the fused item vector. An adversarial subnetwork containing a generator, a true attribute discriminator, and a promiscuous discriminator is constructed. The input is a fused item vector, and the output is a decoupled obfuscated feature. The decoupled obfuscated features are clustered, with the center of each cluster as an obfuscation factor. All obfuscation factors are concatenated to form an obfuscation dictionary matrix. The fused item vector is used as the query vector, and the obfuscation dictionary matrix is ​​used as the key and value. Causal bias terms are obtained through attention-weighted aggregation. Step 4. Prediction and Bias Correction The user aggregated semantic features and the fused item vector are concatenated and input into the prediction head to obtain the initial prediction features. The initial prediction features are then weighted and corrected using a causal bias term to obtain the item prediction score after bias correction. Step 5. Phased Joint Training Based on the decoupling and confusion features of the generator output, an adversarial decoupling loss is constructed; based on the item prediction scores after bias correction, a recommendation ranking loss is constructed; the entire network is iteratively trained until convergence using the composite loss function composed of the adversarial decoupling loss and the recommendation ranking loss; the training is divided into a warm-up phase and a formal adversarial phase: in the warm-up phase, only the parameters of the dual discriminator are updated, while the parameters of other networks are frozen; in the formal adversarial phase, the parameters of the discriminator and other networks are updated alternately.

[0007] The preferred step 1 is as follows: The process involves collecting image and text data of the items. The image data is processed by VGG16 to extract the original visual feature vector, and the text data is processed by Transformer to extract the original text feature vector. The original visual and text feature vectors are then processed by a linear mapping layer with a bias term to obtain the item's visual and text features with consistent dimensions.

[0008] The preferred step 2 is as follows: Step 2.1 First, for the visual features and text features of the object, execute steps 2.1.1 to 2.1.5 in parallel through each modal branch to obtain the visual graph augmentation modal features and the text graph augmentation modal features; Step 2.1.1 Similarity Calculation By calculating the similarity of each pair of all elements, an initial adjacency matrix is ​​constructed, and then the modality graph is obtained. Step 2.1.2 The graph sparsity is adaptively obtained by calculating the signal-to-noise ratio of the modal graph and normalizing it by the maximum value of all signal-to-noise ratios. Step 2.1.3 Perform dynamic sparsity processing on the initial adjacency matrix based on graph structure sparsity to obtain a sparse adjacency matrix; Step 2.1.4 After normalizing the sparse adjacency matrix, perform a residual connection with the input features. Control the weight of the residual connection through hyperparameters to obtain the normalized sparse adjacency matrix. Step 2.1.5 uses a gating mechanism to embed the user ID and fuse it with the input features, which is then used as the input of the 0th layer of the corresponding graph convolutional network. The normalized adjacency matrix is ​​used as the constraint condition for the propagation of each layer of the graph convolutional network, and the layer-by-layer graph convolutional feature propagation is performed from the 1st layer to the Lth layer. After the propagation of the Lth layer is completed, the layer-by-layer mean pooling is performed on the output of the layer to obtain the graph-enhanced modal features. Step 2.2 Input the visual graph augmented modal features and the text graph augmented modal features into the attention fusion module, automatically learn the weight coefficients of the two modalities, and output the fused item vector after weighted aggregation; Step 2.3 Iterate through the set of user's historical interaction items, embed and concatenate the fused item vector with the corresponding user identifier, calculate the weight coefficient of each interaction item through the attention mechanism, and after normalization, sum the weighted item features to obtain the user aggregate semantic features representing user preferences. In the preferred embodiment, the adversarial decoupling loss in step 5 is calculated from the outputs of the true attribute discriminator and the hybrid discriminator, respectively. The true attribute discriminator takes the decoupling and confusion features as input, outputs the attribute prediction value, and calculates the cross-entropy sub-loss based on the attribute prediction value and the true attribute label of the item. The hybrid discriminator takes the decoupling and confusion features as input, outputs the bias prediction value, and calculates the mean square error sub-loss based on the bias prediction value and the uniform distribution constraint term. The adversarial decoupling loss is obtained by superimposing the two sub-losses. The recommendation ranking loss is obtained based on the item prediction score after bias correction.

[0009] The preferred steps of step 2.1.3, which involves dynamic sparsity processing of the initial adjacency matrix based on graph structure sparsity, are as follows: Determine the total number of item nodes participating in the processing; multiply the graph structure sparsity by the total number of item nodes to obtain a threshold for the number of valid neighbors allowed for each item node; obtain the similarity score between each item node and all other item nodes and sort them in descending order; select item nodes whose similarity ranking meets the threshold as valid neighbors; generate a binary mask matrix based on the correspondence between each item node and its valid neighbors; perform element-wise matching operations on the initial similarity adjacency matrix and the binary mask matrix to obtain a sparse adjacency matrix.

[0010] Compared with the prior art, the present invention has the following significant technical effects: 1. This invention introduces a causal inference framework into a multimodal recommendation scenario for the first time. By using a backdoor adjustment criterion, it accurately identifies and eliminates false associations between multimodal features and interaction results caused by confusion factors, allowing the model to truly learn the causal associations corresponding to the real attractiveness of items, and significantly improving the accuracy and fairness of recommendation results.

[0011] 2. This invention introduces a dynamic sparsity strategy based on modal signal-to-noise ratio to adaptively determine the edge density of the graph structure, suppress noise interference caused by unreliable connections, and at the same time retain the effective feature signals of the nodes themselves, thereby alleviating the degradation problem of the depth graph model and significantly improving the robustness and information preservation ability of multimodal graph learning.

[0012] 3. This invention designs a user ID-guided gated attention fusion mechanism, which uses the collaborative signals generated by user embedding as a basis to accurately filter and aggregate the modal features most relevant to the user's current interests, thereby achieving differentiated fine-grained fusion of different modal features and further enhancing the accuracy of user preference representation.

[0013] 4. This invention adopts a phased end-to-end joint training strategy, and uses a composite loss function to collaboratively optimize the bias removal and ranking tasks. While ensuring the independence of feature decoupling, it improves the model's generalization ability and maintains stable and good output performance in various multimodal scenarios.

[0014] This invention has been validated on Amazon's public benchmark datasets. Experimental results show that it improves the Recall@20 metric by approximately 63.4% on the Baby dataset and by approximately 81.7% on the Sports dataset compared to the existing MMGCN method, demonstrating the effectiveness of the denoising and causal bias removal strategy. Attached Figure Description

[0015] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0016] Figure 2This is a flowchart of step 1 in an embodiment of the present invention.

[0017] Figure 3 This is a flowchart of step 2 in an embodiment of the present invention.

[0018] Figure 4 This is a flowchart of step 3 in an embodiment of the present invention.

[0019] Figure 5 This is a flowchart of step 4 in an embodiment of the present invention. Detailed Implementation

[0020] The present invention provides a method for constructing a bias-removed model for multimodal recommendation, as follows: Figure 1 As shown, proceed with the following steps in sequence: Step 1. Input Features and Alignment Acquire visual, textual, and user interaction data of items, unify the dimensions of visual and textual features, and obtain item visual and textual features of the same dimension; specifically as follows: Input features: Let the user set be ， The collection of items is Historical user-item interactions form an implicit feedback matrix. ,in Indicates user With items Observable interactions (such as clicks, purchases, and favorites) occurred between them. This indicates that the interaction does not exist; apart from the interaction data, each item It also includes multimodal features, namely image data (such as images) and text data (such as titles and comments) derived from the items.

[0021] Feature alignment: Examples of embodiments of the present invention Figure 1 As shown: The image data is processed using VGG16 to extract the original visual feature vector, and the text data is processed using Transformer to extract the original text feature vector. The process is as follows: ; ;

[0022] in, Represents the original visual feature vector. Represents the feature vector of the original text; because and Since the dimensions are different and the semantic spaces are not aligned, the original visual feature vectors and original text feature vectors need to be processed by a linear mapping layer with a bias term to obtain the item visual features and item text features with consistent dimensions.

[0023] ;

[0024] in, It is an item The mapped modal feature vector, It is modal The dedicated projection layer weight matrix will transform the original features Mapped to 3D space; It is an item The original modal feature vector. When m=v, it represents the visual features of the object. When m=t, the text features of the item are... It is the projection layer bias vector specific to mode m; Step 2. Multimodal graph preference fusion A unimodal graph is constructed based on the visual and textual features of the items. The graph is then aggregated and fused across modalities using a graph neural network constrained by a normalized sparse adjacency matrix to obtain a fused item vector. For the fused item vectors that have been interacted with by the user, the user identifier is combined with attention-weighted aggregation to output the aggregated semantic features of the user. Specific steps are as follows: Figure 2 As shown: Step 2.1 First, for the visual features and text features of the object, execute steps 2.1.1 to 2.1.5 in parallel through each modal branch to obtain the visual graph augmentation modal features and the text graph augmentation modal features; Step 2.1.1 Similarity Calculation An initial adjacency matrix is ​​constructed by calculating the pairwise similarity of all elements, thereby obtaining the modality graph. Specifically, exponentially enhanced cosine similarity is used to amplify feature differences and highlight significant proximity relationships. The calculation process is shown in the following formula: ;

[0025] in, Let be the similarity value between items a and b in modality m. This value is an element used to construct the adjacency matrix. It is the modal feature vector of item a. It is the modal feature vector m of item b, with temperature parameters. Used to control the sharpness of the similarity distribution: The larger the value, the more uniform the distribution. The smaller the value, the more the high similarity terms are amplified, and the more the low similarity terms are compressed. It is an exponential function used to amplify similarity and highlight significant proximity relationships; Step 2.1.2 The graph sparsity is adaptively obtained by calculating the signal-to-noise ratio of the modal graph and normalizing it by the maximum value of all signal-to-noise ratios. To improve the robustness of the graph structure to noise, a dynamic sparsity strategy is adopted. The sparsity of each modality graph is adaptively determined by the signal-to-noise ratio (SNR) of that mode, and the calculation formula is shown below: ;

[0026] in, It is the graph structure sparsity (or edge density) of mode m. The higher the value, the greater the edge density that can be retained; The basic sparsity constant represents a baseline edge density; It is the signal-to-noise ratio of mode m, used to measure the reliability of the signal in that mode; It is the maximum signal-to-noise ratio among all modes, used for normalization; It is the normalized signal-to-noise ratio, which determines the sparsity weight that a mode should have relative to other modes; Step 2.1.3 Perform dynamic sparsity processing on the initial adjacency matrix based on graph structure sparsity to obtain a sparse adjacency matrix. The specific steps are as follows: Step 2.1.3.1 Determine the total number N of item nodes participating in the processing, and adjust the sparsity of the graph structure. Multiply by the total number of item nodes to obtain the threshold for the number of valid neighbors that each item node is allowed to retain. k ; Step 2.1.3.2 Obtain each item node i The similarity scores with all other item nodes are calculated and sorted in descending order. Items whose similarity ranking meets the stated quantity threshold are selected. k The item node is used as a valid neighbor; Step 2.1.3.3 Based on the correspondence between each item node and its effective neighbors, generate a binary mask matrix, i.e., if the item node... j Belongs to item node i The former k If there are 1 valid neighbor, then let the corresponding element of the mask matrix be... Otherwise, let ; Step 2.1.3.4 Perform an element-wise Hadamard product matching operation on the initial similarity adjacency matrix and the binary mask matrix. Through this operation, the weights of weak connection edges (noisy edges) that do not meet the threshold condition are reset to zero, resulting in a sparse adjacency matrix. Step 2.1.4 After normalizing the sparse adjacency matrix, perform a residual connection with the input features. Control the weight of the residual connection through hyperparameters to obtain the normalized sparse adjacency matrix. To stabilize training and suppress feature degradation and oversmoothing in graph convolution, a novel normalization-regularization operation is introduced: the normalized sparse adjacency matrix is ​​residually concatenated with the input features of this branch, and then determined by the hyperparameters. By controlling the residual weights, a normalized sparse adjacency matrix is ​​obtained. This ensures information aggregation while preserving the node's own signal, preventing excessive smoothing; the calculation formula is as follows: ;

[0027] in, Let be the normalized sparse adjacency matrix of mode m, which serves as the propagation constraint matrix of GCN; These are hyperparameters used to control the weights of residual connections, balancing aggregated information with the preservation of original node information. It is a modal m-sparse adjacency matrix The symmetric normalized form, i.e. For the corresponding mode m The degree matrix is ​​usually A diagonal matrix obtained by summing along rows or columns; The identity matrix is ​​used to implement residual connections (self-connections); this form effectively preserves the initial node features while aggregating neighborhood information, alleviates the degradation problem in deep graph models, and ensures that the graph learning process is more stable and efficient. Step 2.1.5 uses a gating mechanism to fuse the user ID embedding with the input features, which is then used as the input to the 0th layer of the corresponding Graph Convolutional Network (GCN); as shown below: ;

[0028] in, It is an item i In modality m Below, the input features of GCN layer 0 after initialization by the gating mechanism; User u The basic ID embedding represents the collaborative signal; It is an item The mapped modal feature vector; It is the Sigmoid activation function, used to compress the output value to the range (0, 1) as a gating signal; , Shared gate layer weights and biases; ⊙ is the Hadamard product (element-wise product), facilitating gated signals. adjust Information flow; Then, the normalized adjacency matrix is ​​used. As a constraint for the propagation of each layer in the graph convolutional network, the layer-by-layer graph convolutional feature propagation from layer 1 to layer L is performed as follows: ;

[0029] in, It is an item i In modality m Next, GCN Layer feature representation; It is modal m The normalized sparse adjacency matrix, It is an item i In modality m Next, GCN Layer feature representation; Pending L After layer propagation is complete, hierarchical mean pooling is performed on the layer output to obtain graph-enhanced modal features; represented as follows: ;

[0030] in, It is an item i The final graph-enhanced feature representation under mode m; L This is the total number of layers in the GCN; It is GCN. Layer feature representation; It is an average of the embeddings of all layers from layer 0 to layer L; Step 2.2 Input the visual graph augmentation modal features and the text graph augmentation modal features into the attention fusion module, automatically learn the weight coefficients of the two modalities, and output the fused item vector after weighted aggregation; represented as follows: ;

[0031] in, It is an item i The fused item vector; These are the adaptive weights of mode m, learned through an attention mechanism; It is an item i In modality m The following graph shows enhanced feature representation; It involves weighted summation and fusion of graph enhancement features across all modalities; Step 2.3 Iterate through the user's historical interaction item set, embed and concatenate the fused item vector with the corresponding user identifier, calculate the weight coefficient of each interaction item through an attention mechanism, and after normalization, sum the item features by weight to obtain the user aggregate semantic features representing user preferences; represented as follows: ;

[0032] in, User u User aggregation semantic features; It is a collection of items that user u has interacted with; It is an item j The fused item vector; User u Basic ID embedding; It is the query vector for the attention mechanism, used to calculate weights; It's a vector concatenation operation that merges the item vectors. Embedded with user ID Post-concatenation input attention mechanism; It is the Softmax function, which calculates attention weights and assigns importance scores to different neighbors (items) for the user; Step 3. Calculate the causal bias term of multimodal item features based on the fused item vector. To formally analyze bias, a structural causal model (SCM) is used to model the recommendation process. The following nodes represent variables, and arrows indicate direct causal relationships. (1) Traditional likelihood estimation ; (2) Causal intervention ; Where V represents the image modality of the item. Y represents the user comment text modality, Y represents the user interaction result (such as clicks or ratings), and M represents the multimodal feature representation generated by the graph learning module. Z represents the confounding factor, which refers to the potential factors that simultaneously affect both item characteristics and user outcomes. These factors shape the observation modality (M) and, independently of the user's true interests, increase its exposure probability (Y). By performing the do-operation, the confounding influence of inherent object attributes on visual / textual features is severed, thereby separating a pure causal path: image / text → fused representation → interaction.

[0033] The cause-effect diagram reveals two different paths: (1) Causal path (M → Y): reflects the true preference matching, that is, the specific semantic content of the item determines the user's preference.

[0034] (2) Confusion Path (M ← Z → Y): Introducing bias. For example, high-quality promotional images (Z) often bring attractive visual features (M) and high click-through rates (Y); if the model only considers... Learning can mistakenly attribute high interaction rates entirely to visual features, while ignoring the confounding factor of "promotion."

[0035] To formally address the bias problem, we first define key causal factors. The main source of bias is the confounding factor (Z), which represents the latent patterns within an item's multimodal features that are correlated with both the "feature" and the "outcome," but have no causal relationship with actual user preferences. For example, brand names and promotional keywords in text, or background colors and celebrity endorsements in images, create spurious correlations.

[0036] Traditional recommender systems estimate the observed distribution. This distribution is subject to dependencies The deviation is caused by pollution: ;

[0037] To eliminate confounding bias, the goal is to estimate causal effects. , The operator simulates a kind of "physical intervention": forcibly fixing feature M, thereby severing the chaotic link Z → M in the causal graph. According to the backdoor adjustment criterion, this can be achieved by stratifying the confusion factor Z and removing biased conditional priors. Replace with unbiased marginal priors To identify this causal effect: ; By analyzing the confusion factor in its natural distribution By applying marginalization downwards, the backdoor path is blocked, forcing the model to learn only the intrinsic attraction between item features and user preferences. Since Z is continuous and high-dimensional, the above formula can be directly applied... The computational complexity of summation / integration is enormous. Therefore, a normalized weighted geometric mean (NWGM) approximation strategy is adopted to transform theoretical interventions into practical applications. Figure 3 The following three steps are shown: Step 3.1 Construct an adversarial subnetwork containing a generator, a true attribute discriminator, and a promiscuous discriminator. The input is a fused item vector, and the output is a decoupled and obfuscated feature; represented as... ;

[0038] in, It is a decoupling and confusion feature; It is a generator function, usually a neural network; It is the input to the generator, specifically the fused item vector. , These are the parameters of the generator G; Step 3.2: Decoupled confusion features are clustered using K-means, with the center of each cluster serving as a confusion factor. All confusion factors are concatenated to form a confusion dictionary matrix, as detailed below: Based on decoupling and confusion features All items are divided into groups using K-means. There are mutually exclusive clusters, where items within the same cluster have similar intrinsic properties (such as being of the same class or style). For each cluster, a cluster prototype is calculated as a representation of the confounding factor shared by all items within that cluster. Then construct a hierarchical dictionary This provides a discrete space for subsequent summation. Specifically, it decouples the scrambling features. After clustering, the first Cluster prototype Take the mean of the decoupling confusion features of this cluster: ;

[0039] in, It is the first k The cluster prototype of a cluster, that is, the confounding factor representation of that cluster; It is the first k The number of items in each cluster; Items within the cluster j The decoupling and confusion features; It is the average of the decoupling and confusion features of all members within the cluster; Step 3.3 uses the fused item vector as the query vector and the confusion dictionary matrix as the key and value, and obtains the causal bias term through attention-weighted aggregation; specifically as follows: The NWGM (Normalized Weighted Geometric Mean) is used to approximate causal effects with geometric mean, allowing causal interventions to be injected as additive bias terms into the final prediction layer. ; This represents the expected bias introduced by the confounding factor, using the fused item vector. As a query vector, for a mixed dictionary We perform attention calculations to obtain this bias: ;

[0040] in, It is the expected bias term introduced by the confounding factors; K is the total number of confounding factors (i.e., the total number of clusters). It is the first k One confounding factor; It is the attention weight, representing the current item i Subject to k The extent to which each confounding factor has an effect; It is the first k The prior probability of each confounding factor is estimated as the size proportion of the corresponding cluster. ; , It is the projection matrix, used to map the query vector and key vector to the attention space; It is an item i The fused item vector is used as the query vector Q; It is the Softmax function, used to calculate attention weights. , It is a scaling factor used to scale the dot product attention to prevent the gradient from becoming too large.

[0041] Step 4. Prediction and Bias Correction like Figure 4 The user aggregated semantic features and fused item vectors are concatenated and input into the prediction head to obtain initial prediction features. Then, a causal bias term is used to weight and correct the initial prediction features, resulting in a debiased item prediction score. The details are as follows: To generate predictions, first for each user ( ) and items ( Constructing the final, comprehensive representation involves embedding their underlying IDs, which contain the underlying cooperative signals. With the rich semantic representation learned through a complete multimodal processing flow The process involves fusion, combining collaborative signals and semantic representations: ; in, User u The final and comprehensive representation; It is an item i The final and comprehensive representation; , User u and items i Basic ID embedding (cooperative signal); User u Aggregate semantic feature representation; It is an item i The fused item vector; + is the summation operation, which fuses the cooperative signal and semantic representation.

[0042] The resulting final user representation and final item representation are concatenated and first fed into a multilayer perceptron (MLP) prediction head to calculate a basic interaction score. Subsequently, through an additive fusion operation, this basic interaction score is combined with an expected bias correction term introduced by confounding factors. The scores are added together to obtain the final prediction score, which includes causal bias correction; that is, the item prediction score after bias correction. ; in, Useru For items i The item prediction score after bias correction; It is a multilayer perceptron prediction head; User u and items i The final representation of the splicing; It is a causal bias correction term. This is the expected bias term. This is the weight matrix; + indicates additive fusion, which adds the MLP prediction result to the causal bias correction term. This architecture ensures that the rich information captured by all preceding modules is fully utilized to generate the final recommendation results.

[0043] Step 5. Phased Joint Training Based on the decoupling and confusion features of the generator output, an adversarial decoupling loss is constructed; based on the item prediction scores after bias correction, a recommendation ranking loss is constructed; and a composite loss function is formed by the adversarial decoupling loss and the recommendation ranking loss. Iteratively train the entire network until it converges; ;

[0044] in, It is the final composite loss function of the model; It is the Bayesian Personalized Ranking (BPR) loss, the loss function for the main recommendation task; It is an adversarial decoupling loss, an auxiliary task, used for feature decoupling; These are hyperparameters used to balance the weights of the auxiliary loss and the primary recommendation objective; It is an L2 regularization term. These are all the learnable parameters of the model, used to prevent overfitting; The adversarial decoupling loss is calculated from the outputs of the true attribute discriminator and the hybrid discriminator, respectively. The true attribute discriminator takes the decoupling and confusion features as input, outputs the attribute prediction value, and calculates the cross-entropy sub-loss based on the attribute prediction value and the true attribute label of the item. The hybrid discriminator takes the decoupling and confusion features as input, outputs the deviation prediction value, and calculates the mean square error sub-loss based on the deviation prediction value and the uniform distribution constraint term. The adversarial decoupling loss is obtained by superimposing the two sub-losses. ;

[0045] in, It is used to counteract decoupling loss and optimize the generator G; It is cross-entropy loss; It is a true attribute discriminator. These are labels that predict the true attributes (e.g., category) of an item; It is the decoupling and obfuscation feature of the generator output; It is the mean squared error loss. It is a promiscuous discriminator; It is a uniform distribution, where C is the number of classes, and the goal is to minimize... With uniformly distributed MSE loss, it forces Decoupled from user behavior; The recommendation ranking loss is obtained based on the item prediction score after bias correction; The training is divided into a warm-up phase and a formal adversarial phase: in the warm-up phase, only the parameters of the dual discriminator are updated, while other network parameters are frozen; in the formal adversarial phase, the discriminator and other network parameters are updated alternately. The purpose is to prevent the generator G from fitting too quickly in the early stages, which could lead to mode collapse. Specifically, the operation is as follows: (1) Discriminator Independent Update Phase: Freeze the parameters of all graph learning modules (including GCN) and generator G, and update the discriminator independently based solely on the adversarial decoupling loss. Calculate the gradient and update the discriminator separately. and The network weights are adjusted to continuously improve the ability to distinguish between confusing features and interaction signals; (2) Backbone network joint update phase: Freeze the discriminator and The parameters are based on the aforementioned composite loss function. The gradient is jointly computed and the learnable parameters of the generator G and all graph structure fusion and prediction modules are updated. .

[0046] Through the aforementioned alternating update mechanism, the generator G's ability to extract decoupled features dynamically evolves as the multimodal graph representation improves, eventually reaching Nash equilibrium in the game, thus ensuring the purity of feature decoupling and the accuracy of recommendation prediction results.

[0047] In this embodiment of the invention, widely used benchmark datasets from the Amazon Reviews collection are selected for evaluation. These datasets not only contain implicit user-item interactions but also include rich multimodal features.

[0048] The datasets employed a standard preprocessing workflow: a 5-core filter was performed on each dataset, retaining only users and items with at least five interactions. Specifically, two datasets, Baby and Sports, were used, and their data descriptions are shown in Table 1. Table 1 Data Description Table

[0049] The multimodal feature extraction method is as follows: Visual features: For each item, a 4096-dimensional feature vector is extracted from its image using a pre-trained VGG16 network.

[0050] Text features: Using the sentence-transformers library, a 384-dimensional feature vector is generated from the item's metadata (name, title, description, category).

[0051] Model hyperparameter settings: Uniform mapping dimension d=64, number of graph convolutional network layers L=2, residual weight λ=0.1, adversarial decoupling loss balance coefficient λ1=0.2, basic sparsity K. base Determined based on validation set optimization.

[0052] The comparison methods include: BPR, LightGCN, VBPR, MMGCN, GRCN, DualGNN, SLMRec, BM3, MGCN, FREEDOM, SMORE, TMLP and other existing mainstream multimodal recommendation models.

[0053] The experimental results are shown in Tables 2 and 3: Table 2 Comparison of the bias-reduced model with other baseline models in the Baby dataset

[0054] Table 3 Comparison of the bias-reduced model with other baseline models in the Sports dataset

[0055] Results analysis: Experimental results on all benchmark datasets show that the performance of this invention is consistently significantly better than existing baseline models, especially in terms of substantial improvements in the recommendation ranking metrics Recall@20 and NDCG@20.

[0056] On the Baby dataset, this invention achieves a Recall@20 score of 0.1005 and an NDCG@20 score of 0.0424, maintaining strong competitiveness with state-of-the-art SMORE and TMLP models. More importantly, compared to MMGCN (Recall@20: 0.0615) which uses direct feature fusion, this model achieves a significant improvement of approximately 63.4%, fully demonstrating the substantial gains brought by the denoising and debiasing strategies.

[0057] On the Sports dataset, the model's competitiveness was solidified and even enhanced, with a Recall@20 of 0.1099 and an NDCG@20 of 0.0486. On this dataset, the model's performance improvement over MMGCN reached approximately 81.7%, and the difference from the best-performing TMLP model (Recall@20: 0.1152) was minimal.

[0058] Model comparison results confirm that baseline models employing direct feature fusion (such as MMGCN) are extremely sensitive to noise in the original multimodal data and exhibit poor performance. While more advanced methods such as GRCN, MGCN, and FREEDOM achieve better results through indirect fusion or explicit denoising strategies, they still fail to fully utilize the complementarity between different modalities, i.e., they fail to fundamentally address the bias caused by confusion factors.

[0059] This invention first effectively suppresses noise through dynamic sparse graph learning, and then explicitly addresses the causal bias problem through a causal inference module. This dual approach, integrating multimodal denoising and causal debiasing, successfully shifts the recommendation paradigm from maximizing the likelihood of observed data to estimating the intervention effect plus counterfactual inference. This effectively cuts off the confusion path, ensures that the representations learned by the model are invariant to confusion factors, and ultimately significantly improves recommendation accuracy, generalization ability, and the overall robustness of the model.

Claims

1. A method for constructing a bias-free model for multimodal recommendation, characterized in that... Follow these steps in sequence: Step 1. Input features and alignment Acquire visual, textual, and user interaction data of items, unify the dimensions of visual and textual features, and obtain item visual and textual features of the same dimension; Step 2. Multimodal graph preference fusion A unimodal graph is constructed based on the visual and textual features of the items. The graph is then aggregated and fused across modalities using a graph neural network constrained by a normalized sparse adjacency matrix to obtain a fused item vector. For the fused item vectors that have been interacted with by the user, the user identifier is combined with attention-weighted aggregation to output the aggregated semantic features of the user. Step 3. Calculate the causal bias term of multimodal item features based on the fused item vector. An adversarial subnetwork containing a generator, a true attribute discriminator, and a promiscuous discriminator is constructed. The input is a fused item vector, and the output is a decoupled obfuscated feature. The decoupled obfuscated features are clustered, with the center of each cluster as an obfuscation factor. All obfuscation factors are concatenated to form an obfuscation dictionary matrix. The fused item vector is used as the query vector, and the obfuscation dictionary matrix is ​​used as the key and value. Causal bias terms are obtained through attention-weighted aggregation. Step 4. Prediction and Bias Correction The user aggregated semantic features and the fused item vector are concatenated and input into the prediction head to obtain the initial prediction features. The initial prediction features are then weighted and corrected using a causal bias term to obtain the item prediction score after bias correction. Step 5. Phased Joint Training Based on the decoupling and confusion features of the generator output, construct an adversarial decoupling loss; Based on the item prediction scores after bias correction, a recommendation ranking loss is constructed; The entire network is iteratively trained until convergence using the composite loss function consisting of the adversarial decoupling loss and the recommendation ranking loss. The training is divided into a warm-up phase and a formal adversarial phase: in the warm-up phase, only the parameters of the dual discriminator are updated, while the parameters of other networks are frozen; in the formal adversarial phase, the parameters of the discriminator and other networks are updated alternately.

2. The method for constructing a bias-removing model for multimodal recommendation according to claim 1, characterized in that... Step 1 is described in detail as follows: The process involves collecting image and text data of the items. The image data is processed by VGG16 to extract the original visual feature vector, and the text data is processed by Transformer to extract the original text feature vector. The original visual and text feature vectors are then processed by a linear mapping layer with a bias term to obtain the item's visual and text features with consistent dimensions.

3. The method for constructing a bias-removed model for multimodal recommendation according to claim 2, characterized in that... Step 2 is described in detail below: Step 2.1 First, for the visual features and text features of the object, execute steps 2.1.1 to 2.1.5 in parallel through each modal branch to obtain the visual graph augmentation modal features and the text graph augmentation modal features; Step 2.1.1 Similarity Calculation By calculating the similarity of each pair of all elements, an initial adjacency matrix is ​​constructed, and then the modality graph is obtained. Step 2.1.2 The graph sparsity is adaptively obtained by calculating the signal-to-noise ratio of the modal graph and normalizing it by the maximum value of all signal-to-noise ratios. Step 2.1.3 Perform dynamic sparsity processing on the initial adjacency matrix based on graph structure sparsity to obtain a sparse adjacency matrix; Step 2.1.4 After normalizing the sparse adjacency matrix, perform a residual connection with the input features. Control the weight of the residual connection through hyperparameters to obtain the normalized sparse adjacency matrix. Step 2.1.5 uses a gating mechanism to embed the user ID and fuse it with the input features, which is then used as the input of the 0th layer of the corresponding graph convolutional network. The normalized adjacency matrix is ​​used as the constraint condition for the propagation of each layer of the graph convolutional network, and the layer-by-layer graph convolutional feature propagation is performed from the 1st layer to the Lth layer. After the propagation of the Lth layer is completed, the layer-by-layer mean pooling is performed on the output of the layer to obtain the graph-enhanced modal features. Step 2.2 Input the visual graph augmented modal features and the text graph augmented modal features into the attention fusion module, automatically learn the weight coefficients of the two modalities, and output the fused item vector after weighted aggregation; Step 2.3 Iterate through the set of user's historical interaction items, embed and concatenate the fused item vector with the corresponding user identifier, calculate the weight coefficient of each interaction item through the attention mechanism, and after normalization, sum the weighted item features to obtain the user aggregate semantic features representing user preferences. The method for constructing a bias-free model for multimodal recommendation according to claim 3 is characterized in that the adversarial decoupling loss in step 5 is calculated by the output results of the true attribute discriminator and the hybrid discriminator, respectively. The true attribute discriminator takes decoupled confusion features as input and outputs attribute prediction values. It calculates the cross-entropy sub-loss based on the attribute prediction values ​​and the true attribute labels of the items. The hybrid discriminator takes decoupled confusion features as input and outputs bias prediction values. It calculates the mean square error sub-loss based on the bias prediction values ​​and the uniform distribution constraint term. The two sub-losses are superimposed to obtain the adversarial decoupling loss. The recommendation ranking loss is obtained based on the item prediction scores after bias correction.

4. The method for constructing a bias-removed model for multimodal recommendation according to claim 4, characterized in that... The specific steps of step 2.1.3, which involves dynamic sparsity processing of the initial adjacency matrix based on graph structure sparsity, are as follows: Determine the total number of item nodes involved in the processing; multiply the graph structure sparsity by the total number of item nodes to obtain a threshold for the number of valid neighbors allowed for each item node; obtain the similarity score between each item node and all other item nodes and sort them in descending order; select item nodes whose similarity ranking meets the threshold as valid neighbors; generate a binary mask matrix based on the correspondence between each item node and its valid neighbors; perform element-wise matching operations on the initial similarity adjacency matrix and the binary mask matrix to obtain a sparse adjacency matrix.

Images