A multi-modal recommendation method and system based on amplitude calibration contrast alignment

Abstract

The application provides a multi-modal recommendation method and system based on amplitude calibration contrast alignment, comprising: obtaining user-item interaction data and multi-modal features of each item, the multi-modal features comprising at least two of visual features, text features and audio features; encoding each modal feature of each item, and decoupling latent representation into subject semantic embedding and calibration embedding; determining a cross-modal alignment direction by using the subject semantic embedding, estimating cross-modal consistency by using the calibration embedding, generating amplitude weight, constructing amplitude calibration contrast alignment loss, and inhibiting misleading relationships that are similar on the surface but inconsistent across modalities; introducing a user feedback supervision signal, modeling user historical interaction behavior and implicit preferences, and making multi-modal representation consistent with user preference patterns; and fusing the multi-modal representation of the calibrated and aligned items, jointly generating user and item representations, and outputting a recommendation result.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, recommendation systems, and multimodal representation learning, and in particular to a multimodal recommendation method and system based on amplitude calibration contrast alignment. Background Technology

[0002] Multimodal recommendation systems are widely used in real-world online service scenarios such as e-commerce, short video platforms, music platforms, and content distribution platforms. In these scenarios, items typically contain various heterogeneous modal information, including images, text, audio, and structured attributes. Compared to collaborative filtering methods that rely solely on user-item interaction data, multimodal recommendation methods can utilize richer item content information to improve user preference modeling capabilities and recommendation accuracy.

[0003] Existing multimodal recommendation methods typically utilize contrastive learning to achieve semantic alignment between different modalities, enhancing item representation capabilities by bringing similar samples closer together across modal representations. However, most existing methods rely on a uniform similarity signal to determine alignment strength, assuming that sample pairs with higher similarity should be strongly aligned, while ignoring the differences in cross-modal consistency between sample pairs. For samples with only local similarity features but inconsistent overall semantics, traditional methods may still exert a significant optimization influence, leading to incorrect cross-modal alignment. (See attached document) Figure 1 As shown, in the headphone recommendation scenario, the user's historical interaction item is an in-ear Bluetooth headphone, and the candidate items include another in-ear Bluetooth headphone and a pair of over-ear headphones. Since both contain keywords such as "Bluetooth" and "headphones" in their text, traditional similarity-driven alignment methods may incorrectly align the over-ear headphones with the historically interacted item, thus affecting the model's judgment of the user's true preferences. Conversely, if the impact of such locally similar but overall inconsistent sample pairs can be reduced, the in-ear Bluetooth headphone that truly matches the user's preferences can obtain a more accurate representation and recommendation result. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multimodal recommendation method and system based on amplitude calibration comparison alignment, which decouples through multimodal representation; performs multimodal amplitude calibration comparison alignment; and introduces user feedback supervision signals, thereby outputting more accurate recommendation results.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal recommendation method based on amplitude calibration contrast alignment, comprising the following steps:

[0006] S1. Multimodal representation decoupling: Decoupling the main semantic embedding that captures semantic information and the calibration embedding that encodes cross-modal consistency information;

[0007] S2, Multimodal amplitude calibration and contrast alignment; Main semantic embedding determines orientation-aware alignment to maintain semantic similarity; Calibration embedding achieves amplitude-aware calibration to suppress misleading relationships;

[0008] S3. Recommendations are made after incorporating user feedback and supervision signals.

[0009] In a preferred embodiment, the specific process of multimodal characterization decoupling includes the following steps:

[0010] S1.1, For each mode Input features First it is encoded as:

[0011]

[0012] in Representing modes The corresponding encoder.

[0013] S1.2, Latent Representation It is further mapped into two decoupled embeddings:

[0014]

[0015] in The primary semantic embedding is used to capture semantic similarity, while For calibrating the embedding, used to estimate the consistency score;

[0016] S1.3. Introduce a separation loss and achieve decoupling by minimizing the Pearson correlation coefficient between the two decoupled embeddings:

[0017]

[0018]

[0019] in Describing covariance, Indicates standard deviation, This indicates the items being traversed. There are several modalities. Among them, Representing modes Separation loss, This represents the total separation loss obtained by summing the separation losses of each mode; and represents the standard deviation of the main semantic embedding and the calibration embedding, respectively.

[0020] In a preferred embodiment, the specific process of multimodal amplitude calibration contrast alignment includes the following steps:

[0021] S2.1 Given the decoupled representation, first calculate semantic similarity using the main semantic embedding; let Indicates sample In modality The main semantic embedding; for modal pairs ,in Define the alignment value for orientation awareness. for:

[0022]

[0023] in, Indicates sample In modality The main semantic embedding, The cosine similarity score determines the alignment direction between modalities.

[0024] S2.2, Introduce an amplitude sensing mechanism to explicitly adjust the alignment strength; calculate the consistency score using calibration embedding; for modal pairs Defined as:

[0025]

[0026] in This indicates the consistency of cross-modal sample pairs in the calibration space. Indicates sample In modality The calibration embedding below;

[0027] S2.3, Convert the consistency score into alignment magnitude weight:

[0028]

[0029] here, Control the contribution of each cross-modal sample pair to the loss; low-consistency sample pairs will receive a larger contribution. Therefore, they are more strongly pushed away; while highly consistent samples are less affected.

[0030] S2.4, Define the amplitude calibration contrast loss as:

[0031]

[0032]

[0033] in To compare the temperature parameters in the learning process, and Representing different item modalities, Determine the alignment direction. For anchor sample index, For comparison sample indexes that differ from the anchor sample, This represents the orientation-aware alignment value of cross-modal sample pairs from different samples. This represents the orientation-aware alignment value of positive sample pairs across modalities within the same sample. Adaptive adjustment of alignment amplitude Representing mode pairs Amplitude calibration contrast loss, This represents the total alignment loss obtained by summing the losses of all different modal pairs.

[0034] In a preferred embodiment, recommendations are made after incorporating user feedback supervision signals, specifically including the following steps:

[0035] S3.1 Constructing a User-Item Interaction Graph :

[0036]

[0037] The observed interactions constitute the edges in the graph; based on this user-item interaction graph, both the ID embedding and the calibrated main semantic embedding are propagated simultaneously. This represents a user-item interaction graph. Represents a set of users. Let u represent the set of items, i represent the user node, and (u,i) represent an observed interaction edge between user u and item i.

[0038] S3.2, For each mode Modality-aware ID embedding is defined as:

[0039]

[0040]

[0041] in and The ID embeddings for users and items are represented respectively and initialized by a trainable neural network embedding layer; and The diagrams are shown below. The set of neighbors in the middle, and Representing modes Users obtained from the aggregation and items Modality-aware ID embedding; Indicates user Neighbor item nodes, Represents items Neighboring user nodes;

[0042] S3.3. The calibrated main semantic embedding is propagated to obtain a modality-aware feature representation:

[0043]

[0044]

[0045] in and The diagrams are shown below. The set of neighbors in the middle, and Representing users respectively and items In modality Modality-aware feature representation obtained from downpropagation; and These represent neighboring item a and neighboring user b in modal... The main semantic embedding;

[0046] S3.4 Calculate the interaction score in the ID space and multimodal space:

[0047]

[0048]

[0049] in Representing modes User ID embedding below, Indicates user The interaction score of item i in the ID space. This represents the interaction score between the two in the multimodal space;

[0050] S3.5, Corresponding Preference Distribution and Regularization using symmetric KL divergence:

[0051]

[0052] in This indicates user feedback regarding monitoring losses. Indicates the total number of users. and Representing users respectively Preference distribution in ID space and multimodal space;

[0053] S3.6, Use a multi-head self-attention mechanism to fuse modality-aware ID embeddings; Let... For the number of modes, Let h be the number of attention heads; for each attention head h, the query, key, and value projection matrices are respectively... For each mode The fused representation is obtained by performing attention calculations on all modalities:

[0054]

[0055] in Representing modes User ID embedding below, Indicates user In modality The representation after multi-head self-attention fusion;

[0056] S3.7. Employing a mean pooling strategy to fuse multimodal embeddings:

[0057]

[0058] in Indicates user Multimodal fusion ID embedding, Indicates user In modality The representation after attention fusion The total number of modalities is represented, and the multimodal fusion ID of the item is obtained in the same way.

[0059] S3.8. Combine the fused ID embedding with the modality-aware feature embedding:

[0060]

[0061] in These are learnable parameters used to control the contribution of modal features; item embedding. The methods of obtaining them are similar;

[0062] S3.9, User With items The predicted interaction score is defined as:

[0063]

[0064] S3.10. Use Bayesian Personalized Ranking loss to supervise the recommendation task:

[0065]

[0066] Where A is the user Positive sample items and negative sample items The set of triplets formed, Represents the sigmoid function;

[0067] S3.11 The overall objective function is calculated as follows:

[0068]

[0069] hyperparameters and Used for balance and Weights.

[0070] In a preferred embodiment, the model is trained using the Adam optimizer with an initial learning rate of 1e-4, a batch size of 2048, 200 training epochs, and an early stopping strategy; hyperparameters... and Optimize within the range {0.01, 0.1, 1.0, 1.5} using grid search, and compare with the learned temperature parameters. Optimize within the range of [0.1, 1] to determine the number of attention heads. Optimize within the range of {2, 4, 8, 16}.

[0071] The present invention also provides a computer device, comprising:

[0072] The processor, memory, and bus, wherein the memory stores machine-readable instructions executed by the processor;

[0073] When the computer device is running, the processor and the memory communicate via a bus, and the machine-readable instructions are executed by the processor, such as a multimodal recommendation method based on amplitude calibration contrast alignment.

[0074] The present invention also provides a computer-readable storage medium, comprising:

[0075] The computer-readable storage medium contains a computer program;

[0076] The computer program is executed by the processor as described in the multimodal recommendation method based on amplitude calibration contrast alignment.

[0077] This invention also provides a multimodal recommendation system based on amplitude calibration contrast alignment, including a processor, a memory, and a bus. The memory stores machine-readable instructions executed by the processor. When the system is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, they are as described in the multimodal recommendation method based on amplitude calibration contrast alignment.

[0078] Compared with existing technologies, this invention has the following advantages: To evaluate the performance of the method of this invention on recommendation tasks, the performance was verified on the Baby, Sports, and TikTok datasets. The Baby and Sports datasets are from Amazon, where items are represented by 4096-dimensional visual CNN features and Sentence-BERT-based text embeddings, respectively. The TikTok dataset contains three modalities: visual, audio, and text, and its features are extracted by a pre-trained encoder. This invention uses three widely used ranking metrics: Recall, Precision, and NDCG (Normalized Decay Cumulative Gain). Higher metric values ​​indicate better recommendation performance.

[0079] This invention compares with 12 representative baseline methods, including methods without explicit modeling for cross-modal alignment (LightGCN, AutoCF, MCDRec, LRMM, and MoDiCF) and methods with explicit modeling for multimodal alignment (MMSSL, BM3, MG, DiffMM, CI2MG, MILK, and DGMRec). Experiments on multimodal recommendation with random modal missing values ​​were conducted on three datasets, and the results are shown in Table 1. This invention consistently outperforms all baseline methods on the three datasets, demonstrating that the proposed modal decoupling and amplitude calibration-based alignment mechanism effectively improves multimodal recommendation performance and maintains a stable advantage even under the challenging setting of 40% modality missing values. Compared to methods without explicit modeling for cross-modal alignment, this invention effectively alleviates the misleading alignment problem based on local similarity by decoupling modal representations into semantic embeddings and calibration embeddings, and by adaptively adjusting the alignment strength using calibration branches, achieving significant improvements across various metrics. Compared to methods that explicitly model multimodal alignment, this invention further introduces an amplitude calibration mechanism to differentially model the contributions of different cross-modal sample pairs. This results in stronger optimization constraints for truly consistent sample pairs, while effectively suppressing low-consistency samples, thereby learning a more compact and discriminative multimodal representation. On datasets with richer multimodal information and more complex interactions, such as TikTok, this invention achieves a Recall@20 improvement of up to 9.95%, demonstrating stronger cross-modal semantic modeling capabilities and generalization performance.

[0080] Table 1. Experimental results (%) for the three datasets with a modality missing rate of 40%.

[0081] (Best performance is indicated in bold, second-best performance is indicated by underline, and OOM indicates an out-of-memory error during training).

[0082] Attached Figure Description

[0083] Figure 1 This is a schematic diagram illustrating the difference between traditional similarity-driven alignment and amplitude-calibrated alignment in multimodal recommendation. The green polygons represent sample groups that the model tends to align with.

[0084] Figure 2 This is a general framework diagram of the multimodal recommendation method based on amplitude calibration contrast alignment provided in the embodiments of the present invention;

[0085] Figure 3 This is a schematic diagram illustrating an example of how the present invention and a suboptimal performance baseline handle cross-modal inconsistent samples on the Sports dataset. Detailed Implementation

[0086] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0087] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0088] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0089] like Figure 2 As shown, the specific implementation process of the multimodal recommendation method based on amplitude calibration contrast alignment in this embodiment of the invention includes the following steps:

[0090] S1 is used for multimodal representation decoupling; it decouples the main semantic embedding that captures semantic information from the calibration embedding that encodes cross-modal consistency information.

[0091] S2 is used for multimodal amplitude calibration and contrast alignment; the main semantic embedding determines orientation-aware alignment to maintain semantic similarity; the calibration embedding achieves amplitude-aware calibration to suppress misleading relationships;

[0092] S3. Recommendations are made after incorporating user feedback and supervision signals.

[0093] In this embodiment, the specific process of multimodal characterization decoupling includes the following steps:

[0094] S1.1, For each mode Input features First, it is encoded as:

[0095]

[0096] S1.2, Latent Representation This is further mapped into two decoupled embeddings:

[0097]

[0098] in Capture semantic similarity, and Used to estimate consistency scores.

[0099] S1.3 To facilitate clear separation between the two representations, a separation loss is introduced, and decoupling is achieved by minimizing the Pearson correlation coefficient between the two parts:

[0100]

[0101]

[0102] in Describing covariance, Indicates standard deviation, Iterate through all modalities corresponding to the item.

[0103] In this embodiment, the specific process of multimodal amplitude calibration and alignment includes the following steps:

[0104] S2.1 Given the decoupled representation, first calculate semantic similarity using the main semantic embedding. Let Indicates sample In modality The main semantic embedding under the modality pair. (in The alignment value for orientation awareness is defined as follows:

[0105]

[0106] The cosine similarity score determines the alignment direction between modalities.

[0107] S2.2, Further introduce an amplitude-sensing mechanism to explicitly adjust the alignment strength. Calculate the consistency score using calibration embedding. For modal pairs... Defined as:

[0108]

[0109] in This indicates the consistency of cross-modal sample pairs in the calibration space.

[0110] S2.3, Convert the consistency score into alignment magnitude weight:

[0111]

[0112] here, Control the contribution of each cross-modal sample pair to the loss. Low-consistency sample pairs will receive a larger contribution. The high-consistency samples are thus more strongly pushed away, while the high-consistency samples are less affected.

[0113] S2.4, Define the amplitude calibration contrast loss as:

[0114]

[0115]

[0116] in Determine the alignment direction. Adaptive adjustment of alignment amplitude To compare the temperature parameters in the learning process, and Represents different item modalities.

[0117] In this embodiment, recommendations are made after incorporating user feedback supervision signals, specifically including the following steps:

[0118] S3.1 Constructing a user-item interaction graph:

[0119]

[0120] The observed interactions constitute the edges in the graph. Based on this graph, we simultaneously propagate the ID embedding and the calibrated main semantic embedding.

[0121] S3.2 For each modality m, the modality-aware ID embedding is defined as:

[0122]

[0123]

[0124] in and These represent the ID embeddings for users and items, respectively, which are initialized by a trainable neural network embedding layer; and The diagrams are shown below. The set of neighbors in the middle.

[0125] S3.3. The calibrated main semantic embedding is propagated to obtain a modality-aware feature representation:

[0126]

[0127]

[0128] S3.4 To ensure consistency between multimodal representations and collaborative preference patterns, we compute interaction scores in both the ID space and the multimodal space:

[0129]

[0130]

[0131] in Representing modes Embedded user ID.

[0132] S3.5, Corresponding Preference Distribution and Regularization using symmetric KL divergence:

[0133]

[0134] S3.6. Use a multi-head self-attention mechanism to fuse modality-aware ID embeddings to capture cross-modal relevance. Let... For the number of modes, Let be the number of attention heads. For each attention head... The query, key, and value projection matrices are respectively For each mode The fused representation is obtained by performing attention calculations on all modalities:

[0135]

[0136] S3.7. Employing a mean pooling strategy to fuse multimodal embeddings:

[0137]

[0138] S3.8. Combine the fused ID embedding with the modality-aware feature embedding:

[0139]

[0140] in These are learnable parameters used to control the contribution of modal features. Item embedding. The method of obtaining them is similar.

[0141] S3.9, User With items The predicted interaction score is defined as:

[0142]

[0143] S3.10. Use Bayesian Personalized Ranking (BPR) loss to supervise the recommendation task:

[0144]

[0145] in For users Positive sample items and negative sample items The set of triplets formed, This represents the sigmoid function.

[0146] S3.11 The overall objective function is calculated as follows:

[0147]

[0148] hyperparameters and Used for balance and Weights.

[0149] Preferably, in this embodiment, the model training uses the Adam optimizer, with an initial learning rate set to 1e-4, a batch size of 2048, and 200 training epochs. An early stopping strategy is employed (training stops if there is no improvement after 50 consecutive epochs on the validation set Recall@10); hyperparameters... and The temperature parameters can be optimized within the range of {0.01, 0.1, 1.0, 1.5} through grid search, and compared with the learned parameters. Optimize within the range of [0.1, 1] to determine the number of attention heads. Optimize within the range of {2, 4, 8, 16}.

[0150] To verify the discriminative ability of this invention in cross-modal inconsistent scenarios, its recommendation results on the Sports dataset were compared and analyzed with those of the suboptimal performance baseline method. Figure 3As shown, in user Jack's training preference scenario, his historical interests are mainly concentrated on weight training equipment. Two typical samples in the candidate set are analyzed: Candidate C1 is a flat bench press bench, whose visual structure and text description are highly consistent with "bench press bench," belonging to a cross-modal consistent sample; Candidate C2 is an adjustable sit-up bench, whose text contains keywords such as "adjustable," showing high textual surface similarity to the user's interests, but its visual structure differs significantly from the target training device, belonging to a typical cross-modal inconsistent sample. In the baseline method that relies solely on semantic similarity modeling, because modal consistency is not explicitly distinguished, the model is more susceptible to interference from textual keywords, thus incorrectly increasing the recommendation weight of C2, leading to a deviation from the user's true training preferences. This invention introduces a modal decoupling mechanism, separating the semantic representation from the consistency calibration signal for modeling, and using the calibration branch to explicitly constrain cross-modal consistency, enabling the reliability differences between different modalities to be effectively characterized. Although C2 has high text modal similarity, it has low visual modal consistency. Therefore, this invention reduces its cross-modal alignment contribution to suppress its misleading influence in the optimization process. As for C1, since it shows high consistency in both visual and text modalities, this invention enhances its alignment strength to give it a higher recommendation weight.

[0151] This invention can effectively distinguish between interference samples that are “superficially similar but semantically inconsistent” and target samples that are “seriously semantically consistent”, thereby generating a more compact and consistent multimodal representation, significantly improving the accuracy and robustness of recommendation decisions, and enhancing the reliability of the model in complex cross-modal scenarios.

Claims

1. A multimodal recommendation method based on amplitude calibration contrast alignment, characterized in that, Includes the following steps: S1. Multimodal representation decoupling: Decoupling the main semantic embedding that captures semantic information and the calibration embedding that encodes cross-modal consistency information; S2, Multimodal amplitude calibration and contrast alignment; Main semantic embedding determines orientation-aware alignment to maintain semantic similarity; Calibration embedding achieves amplitude-aware calibration to suppress misleading relationships; S3. Recommendations are made after incorporating user feedback and supervision signals.

2. The multimodal recommendation method based on amplitude calibration contrast alignment according to claim 1, characterized in that, The specific process of decoupling multimodal characterization includes the following steps: S1.1, For each mode Input features First, it is encoded as: in Representing modes The corresponding encoder; S1.2, Latent Representation This is further mapped into two decoupled embeddings: in The primary semantic embedding is used to capture semantic similarity, while For calibrating the embedding, used to estimate the consistency score; S1.

3. Introduce a separation loss and achieve decoupling by minimizing the Pearson correlation coefficient between the two decoupled embeddings: in Describing covariance, Indicates standard deviation, This indicates the items that are being traversed. One modality; among which Representing modes Separation loss, This represents the total separation loss obtained by summing the separation losses of each mode; and represents the standard deviation of the main semantic embedding and the calibration embedding, respectively.

3. The multimodal recommendation method based on amplitude calibration contrast alignment according to claim 1, characterized in that, The specific process of multimodal amplitude calibration and alignment includes the following steps: S2.1 Given the decoupled representation, first calculate semantic similarity using the main semantic embedding; let Indicates sample In modality The main semantic embedding; for modal pairs ,in Define the alignment value for orientation awareness. for: in, Indicates sample In modality The main semantic embedding, The cosine similarity score determines the alignment direction between modalities. S2.2, Introduce an amplitude sensing mechanism to explicitly adjust the alignment strength; calculate the consistency score using calibration embedding; for modal pairs Defined as: in This indicates the consistency of cross-modal sample pairs in the calibration space. Indicates sample In modality The calibration embedding below; Indicates sample In modality The calibration embedding below; S2.3, Convert the consistency score into alignment magnitude weight: here, Control the contribution of each cross-modal sample pair to the loss; low-consistency sample pairs will receive a larger contribution. Therefore, they are more strongly pushed away; while highly consistent samples are less affected. S2.4, Define the amplitude calibration contrast loss as: in To compare the temperature parameters in the learning process, and Representing different item modalities, Determine the alignment direction. For anchor sample index, For comparison sample indexes that differ from the anchor sample, This represents the orientation-aware alignment value of cross-modal sample pairs from different samples. This represents the orientation-aware alignment value of positive sample pairs across modalities within the same sample. Adaptive adjustment of alignment amplitude Representing mode pairs Amplitude calibration contrast loss, This represents the total alignment loss obtained by summing the losses of all different modal pairs.

4. The multimodal recommendation method based on amplitude calibration contrast alignment according to claim 1, characterized in that, Recommendations are made after incorporating user feedback and supervision signals, specifically including the following steps: S3.1 Constructing a User-Item Interaction Graph : The observed interactions constitute the edges in the graph; based on this user-item interaction graph, both the ID embedding and the calibrated main semantic embedding are propagated simultaneously. This represents a user-item interaction graph. Represents a set of users. Let u represent the set of items, i represent the user node, and (u,i) represent an observed interaction edge between user u and item i. S3.2, For each mode Modality-aware ID embedding is defined as: in and The ID embeddings for users and items are initialized by a trainable neural network embedding layer, respectively. and The diagrams are shown below. The set of neighbors in the middle, and Representing modes Users obtained from the aggregation and items Modality-aware ID embedding; Indicates user Neighbor item nodes, Represents items Neighboring user nodes; S3.

3. The calibrated main semantic embedding is propagated to obtain a modality-aware feature representation: in and The diagrams are shown below. The set of neighbors in the middle, and Representing users respectively and items In modality Modality-aware feature representation obtained from downpropagation; and These represent neighboring item a and neighboring user b in modal... The main semantic embedding; S3.4 Calculate the interaction score in the ID space and multimodal space: in Representing modes User ID embedding below, Indicates user With items Interaction score in the ID space, This represents the interaction score between the two in the multimodal space; S3.5, Corresponding Preference Distribution and Regularization using symmetric KL divergence: in This indicates user feedback regarding monitoring losses. Indicates the total number of users. and Representing users respectively Preference distribution in ID space and multimodal space; S3.6, Use a multi-head self-attention mechanism to fuse modality-aware ID embeddings; Let... For the number of modes, Let h be the number of attention heads; for each attention head h, the query, key, and value projection matrices are respectively... For each mode The fused representation is obtained by performing attention calculations on all modalities: in Representing modes User ID embedding below, Indicates user In modality The representation after multi-head self-attention fusion; S3.

7. Employing a mean pooling strategy to fuse multimodal embeddings: in Indicates user Multimodal fusion ID embedding, Indicates user In modality The representation after attention fusion The total number of modalities is represented, and the multimodal fusion ID of the item is obtained in the same way. S3.

8. Combine the fused ID embedding with the modality-aware feature embedding: in These are learnable parameters used to control the contribution of modal features; item embedding. The methods of obtaining them are similar; S3.9, User With items The predicted interaction score is defined as: S3.

10. Use Bayesian Personalized Ranking loss to supervise the recommendation task: Where A is the user Positive sample items and negative sample items The set of triplets formed, Represents the sigmoid function; S3.11 The overall objective function is calculated as follows: hyperparameters and Used for balance and Weights.

5. A multimodal recommendation method based on amplitude calibration contrast alignment according to claim 1, characterized in that, The model was trained using the Adam optimizer with an initial learning rate of 1e-4, a batch size of 2048, 200 training epochs, and an early stopping strategy. Hyperparameters... and Optimize within the range {0.01, 0.1, 1.0, 1.5} using grid search, and compare with the learned temperature parameters. Optimize within the range of [0.1, 1] to determine the number of attention heads. Optimize within the range of {2, 4, 8, 16}.

6. A computer device, characterized in that, include: The processor, memory, and bus, wherein the memory stores machine-readable instructions executed by the processor; When the computer device is running, the processor communicates with the memory via a bus, and when the machine-readable instructions are executed by the processor, they perform a multimodal recommendation method based on amplitude calibration contrast alignment as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium contains a computer program; The computer program is executed by the processor to perform a multimodal recommendation method based on amplitude calibration contrast alignment as described in any one of claims 1 to 5.

8. A multimodal recommendation system based on amplitude calibration contrast alignment, comprising a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executed by the processor; characterized in that, When the system is running, the processor communicates with the memory via a bus, and the machine-readable instructions are executed by the processor as described in any one of claims 1 to 5, which is a multimodal recommendation method based on amplitude calibration contrast alignment.

Images