Recommendation method and system based on cross-modal collaboration-semantic alignment

Abstract

The invention discloses a recommendation method and system based on cross-modal collaboration-semantic alignment. The recommendation method comprises the steps of obtaining text attribute data and user-article interaction data in user information and article information; based on the user information text attribute and the article information text attribute, obtaining user semantic embedding and article semantic embedding; performing multi-layer perception deep semantic coding on user semantic embedding and article semantic embedding to generate high-order semantic embedding representation; constructing a user-article bipartite graph based on the user-article interaction data, and introducing a residual connection mechanism, a node attention mechanism and a robust enhancement mechanism to generate collaborative embedding representation; the collaborative embedding representation and the high-order semantic embedding representation are aligned based on a multi-head self-attention mechanism, weighted fusion is performed by using a gating mechanism, fusion embedding is constrained based on a self-supervised learning strategy, combined distillation loss and cross-modal contrast loss, and final fusion embedding representation is generated; and outputting a recommendation result based on the final fusion embedded representation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and recommendation system technology, specifically relating to a recommendation method and system based on cross-modal collaborative-semantic alignment. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] With the widespread adoption of the internet and mobile devices, users generate massive amounts of interactive behaviors and text comments on e-commerce, social media, and other platforms, forming rich collaborative signals and semantic information. On the one hand, collaborative filtering methods effectively uncover group preference patterns by analyzing user-item interaction matrices, but their performance is often limited by data sparsity and the cold-start problem. On the other hand, pre-trained large language models (such as BERT and GPT) demonstrate powerful capabilities in natural language understanding, mapping user and item attribute text or comments into high-dimensional semantic vectors, introducing rich semantic features into recommendation systems.

[0004] To improve the recommendation quality of models, an increasing number of works attempt to fuse collaborative embeddings with semantic embeddings, including simple concatenation, multidimensional weighting, or shared encoding spaces. However, due to significant differences in distribution, scale, and information emphasis between the two types of embeddings, direct fusion is prone to introducing noise and causing information inconsistencies, and the fusion effect is unstable in long-tail items and new user scenarios. Furthermore, collaborative models based on graph neural networks (GNNs) often face problems such as oversmoothing, vanishing gradients, and structural noise when capturing high-order neighbor relationships, further limiting the model's generalization ability and robustness.

[0005] Therefore, effectively aligning and fusing multi-source semantic information while deeply mining collaborative signals, and resisting noise interference and mode shift through structural enhancement and self-supervised optimization mechanisms, has become a core challenge in current recommender system research. Existing methods still have significant room for improvement in modeling multimodal dependencies, optimizing cross-modal information flow, and enhancing model stability in sparse and noisy environments. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a recommendation method and system based on cross-modal collaborative-semantic alignment. By introducing deep semantic encoding, multi-layer graph structure enhancement, cross-modal fusion and self-supervised alignment optimization, it achieves deep integration of collaborative filtering and semantic modeling, thereby improving the performance of the recommendation system in scenarios such as cold start, high noise and data sparsity.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: On the one hand, the technical solution of the present invention provides a recommendation method based on cross-modal collaborative-semantic alignment, including: Retrieve text attribute data and user-item interaction data from user and item information; Based on the text attributes of user information and the text attributes of item information, obtain the semantic embedding of users and the semantic embedding of items; Multi-layer perceptual deep semantic encoding is performed on user semantic embeddings and item semantic embeddings respectively to generate high-order semantic embedding representations; Based on user-item interaction data, a user-item bipartite graph is constructed, and residual connection mechanism, node attention mechanism and robust enhancement mechanism are introduced to generate collaborative embedding representation; Aligning collaborative embedding representations and high-order semantic embedding representations based on multi-head self-attention mechanism, and using gating mechanism for weighted fusion, the final fused embedding representation is generated by combining distillation loss and cross-modal contrastive loss based on self-supervised learning strategy. The recommendation results are output based on the final fused embedding representation.

[0008] In at least one embodiment, the deep semantic encoding process specifically includes:

[0009]

[0010]

[0011] in For user semantic embedding vectors or item semantic embedding vectors; For the first Layer weights and bias coefficients, This is a high-order semantic embedding representation generated through deep semantic encoding.

[0012] In at least one embodiment, the residual connection mechanism is specifically as follows: In the graph structure Each layer performs standard graph convolution on its input node representations to obtain a temporary representation after graph convolution; the input node representations of this layer are then residually fused with the temporary representation after graph convolution to obtain the final representation. Layered embedding representation.

[0013] In at least one embodiment, the node attention mechanism is used to assign attention weights to neighboring nodes and perform feature aggregation based on the importance of neighbors to the target node. Specifically, the features of the target node and its neighboring nodes are concatenated and then linearly mapped and nonlinearly transformed. The attention scores of all neighboring nodes are normalized by softmax operation to obtain the importance distribution coefficients of the neighboring nodes. The representation vector of the target node is then weighted and aggregated with the feature vectors of its neighboring nodes in the next layer to obtain the embedding representation of the target node in the current layer.

[0014] In at least one embodiment, the robustness enhancement mechanism includes an edge sampling operation and an embedding perturbation strategy; wherein, the edge sampling operation is to randomly retain or delete adjacent edges in the user-item bipartite graph according to probability; and the embedding perturbation strategy is to inject Gaussian noise into the node representation during the embedding propagation process.

[0015] In at least one embodiment, the collaborative embedding representation and the higher-order semantic embedding representation are aligned based on a multi-head self-attention mechanism. Specifically, the collaborative embedding representation and the higher-order semantic embedding representation are used as inputs, and multiple attention heads are used to calculate the attention weights of each head in parallel and generate a fused feature representation. The fused feature representations generated by each head are concatenated and mapped back to the original dimension to obtain the attention fusion feature.

[0016] In at least one embodiment, a gating mechanism is used for weighted fusion, specifically: the attention fusion feature and the collaborative embedding representation are concatenated and input into a fully connected layer, and a gating vector is obtained through the Sigmoid activation function; the attention fusion feature and the collaborative embedding representation are weighted and fused based on the gating vector to obtain a multimodal fusion embedding representation.

[0017] In at least one embodiment, the distillation loss is specifically expressed as: ; in, The squared Euclidean distance between the multimodal fusion embedding representation and the high-order semantic embedding representation measures the degree of inconsistency between the two. The weights for the distillation loss are dynamically adjusted based on the cosine similarity between the multimodal fusion embedding representation and the higher-order semantic embedding representation, and are expressed as: ; in, The basic weight for distillation; This is the adjustment coefficient; Cosine similarity between the multimodal fusion embedding representation and the higher-order semantic embedding representation.

[0018] In at least one embodiment, the cross-modal contrast loss includes the original embedding contrast loss and the fused embedding contrast loss; wherein, the original embedding contrast loss is specifically: ; in, For collaborative embedding representation, and For semantic embedding, Positive sample pairs are those with collaborative semantics, meaning they originate from the same user or item; exist When the time is a negative sample pair, that is, a combination of collaboration and semantic mismatch; Represents the cosine similarity function; The temperature coefficient controls the sensitivity of the comparison similarity distribution; a smaller value will amplify the similarity differences. The specific fusion embedding contrast loss is as follows: ; in, For multimodal fusion embedding representation, and For semantic embedding; The cross-modal contrast loss is as follows: ; The total loss function is specifically expressed as: ; in, Loss due to self-supervised learning; Enhanced semantic-co-alignment; Strengthen the discriminative power between semantics and collaboration; and The balancing coefficient for the loss term is dynamically adjusted using the validation set.

[0019] On the other hand, the technical solution of the present invention also provides a recommendation system based on cross-modal collaborative-semantic alignment, including: The data acquisition module is configured to: acquire text attribute data and user-item interaction data from user information and item information; The semantic embedding acquisition module is configured to acquire user semantic embedding and item semantic embedding based on user information text attributes and item information text attributes. The high-order semantic embedding generation module is configured to perform multi-layer perceptual deep semantic encoding on user semantic embedding and item semantic embedding respectively to generate high-order semantic embedding representations; The collaborative embedding generation module is configured to: construct a user-item bipartite graph based on user-item interaction data, introduce residual connection mechanism, node attention mechanism and robust enhancement mechanism, and generate collaborative embedding representation; The fusion embedding generation module is configured to: align collaborative embedding representations and high-order semantic embedding representations based on a multi-head self-attention mechanism, perform weighted fusion using a gating mechanism, and constrain the fusion embedding based on a self-supervised learning strategy, using joint distillation loss and cross-modal contrastive loss to generate the final fusion embedding representation; The recommendation module is configured to output recommendation results based on the final fused embedding representation.

[0020] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) The recommendation method based on cross-modal collaborative-semantic alignment of this invention employs a deep semantic encoding module to perform high-order abstract modeling of user and item attribute text, effectively alleviating the problems of feature distribution shift and overfitting. Through multi-layer nonlinear transformation, the expressive power of semantic embedding is improved, providing basic support for cold-start and heterogeneous attribute recommendation, which is significantly better than recommendation methods that only use collaborative information.

[0021] 2) The recommendation method based on cross-modal collaborative-semantic alignment of this invention introduces a graph structure enhancement mechanism. By combining residual connections, node attention, and embedding perturbation with structural optimization strategies in the user-item interaction graph, this method improves the modeling ability of high-order collaborative relationships and enhances the effectiveness of information propagation paths. This mechanism overcomes the oversmoothing and noise sensitivity problems existing in traditional GCN structures, improving deep-level robustness while maintaining shallow-level discriminative ability.

[0022] 3) This invention proposes a cross-modal fusion mechanism based on cross-modal collaborative-semantic alignment recommendation. It utilizes multi-head attention to align the semantic dimensions of collaborative embeddings and semantic embeddings, and then dynamically adjusts the fusion weight through a gating network, achieving adaptive integration of the two modalities. Compared to fixed fusion strategies, this method offers greater representational flexibility and personalized modeling capabilities, automatically adjusting the fusion strategy according to data distribution.

[0023] 4) This invention also has good structural modularity and expansion compatibility, and can be seamlessly integrated with existing mainstream collaborative filtering models (such as LightGCN, DCCF, GCCF, etc.), significantly improving the recommendation performance of the original model in complex scenarios. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is a schematic diagram of the recommendation method based on cross-modal collaborative-semantic alignment disclosed in Embodiment 1 of the present invention; Figure 2This is a schematic diagram of the overall model framework of the recommendation method based on cross-modal collaborative-semantic alignment disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] As described in the background section, the purpose of this invention is to overcome the shortcomings of the prior art and provide a recommendation method and system based on cross-modal collaborative-semantic alignment. By introducing deep semantic encoding, multi-layer graph structure enhancement, cross-modal fusion, and self-supervised alignment optimization, the invention achieves deep integration of collaborative filtering and semantic modeling, thereby improving the performance of the recommendation system in scenarios such as cold start, high noise, and sparse data.

[0028] Example 1 In a typical embodiment of the present invention, such as Figure 1 As shown, this embodiment discloses a recommendation method based on cross-modal collaborative-semantic alignment, including the following steps: S100. Obtain text attribute data and user-item interaction data from user information and item information; S200. Based on the text attributes of user information and the text attributes of item information, obtain the semantic embedding of the user and the semantic embedding of the item; S300. Perform multi-layer perceptual deep semantic encoding on user semantic embedding and item semantic embedding respectively to generate high-order semantic embedding representation; S400. Construct a user-item bipartite graph based on user-item interaction data, and introduce residual connection mechanism, node attention mechanism and robust enhancement mechanism to generate collaborative embedding representation; S500. Aligns collaborative embedding representations and high-order semantic embedding representations based on multi-head self-attention mechanism, and performs weighted fusion using gating mechanism. Based on self-supervised learning strategy, it combines distillation loss and cross-modal contrastive loss to constrain fusion embedding and generate final fused embedding representation. S600. Output recommendation results based on the final fused embedded representation.

[0029] The following section provides a detailed explanation of the above-mentioned recommended method based on cross-modal collaborative semantic alignment, with reference to specific implementation methods.

[0030] S100. Obtain text attribute data and user-item interaction data from user information and item information.

[0031] In this step, the text attribute data of user information refers to the coherent text generated by a large model such as chatgpt-3.5-turbo, which contains text information that the user has interacted with (such as the titles of items that have been interacted with, the text attribute data of the items, and the user's comments).

[0032] Text attribute data for item information refers to coherent text generated from item text information (such as title, item description, item category attributes, user comments on the item, etc.) through a large model.

[0033] User-item interaction data refers to information about user interactions with items (such as purchasing, browsing, rating, etc.).

[0034] It should be noted that all data acquired in this invention is done in accordance with laws, regulations, and ethical standards, and the data is used legally.

[0035] S200. Based on the text attributes of user information and the text attributes of item information, obtain the semantic embedding of the user and the semantic embedding of the item.

[0036] In this step, a large language model (such as Chatgpt-3.5-turbo, GPT-4, or other pre-trained language models with contextual understanding capabilities) is used to encode the text attribute data of user and item information acquired by S100. Specifically, the user text attribute sequence and the item text attribute sequence are input into the large language model (such as Chatgpt-3.5-turbo, GPT-4, or other pre-trained language models with contextual understanding capabilities) to obtain user semantic embedding vectors and item semantic embedding vectors. express.

[0037] S300. Perform multi-layer perceptual deep semantic encoding on user semantic embedding and item semantic embedding respectively to generate high-order semantic embedding representation.

[0038] To enhance semantic representation capabilities, this embodiment designs a multilayer perceptron (MLP) for user semantic embedding or item semantic embedding. Deep semantic encoding is performed. This multilayer perceptron (MLP) employs a one- to three-layer structure, with each layer including linear transformation, LayerNorm normalization, GELU activation function, and Dropout regularization. Through multi-layer nonlinear mapping and normalization operations, the problems of feature distribution shift and overfitting are effectively mitigated, making semantic features more abstract and possessing stronger generalization ability.

[0039] Specifically, using original user semantic embedding or item semantic embedding The deep semantic encoding process, taking the input as an example, is as follows:

[0040]

[0041]

[0042] in, For the multilayer perceptron The weights and bias coefficients of each layer are used in the multilayer perceptron, which has a three-layer structure. Each layer contains a linear transformation, LayerNorm, GELU, and Dropout, and finally outputs a high-order semantic embedding representation. This data is intended for subsequent cross-modal fusion. By employing deep semantic encoding to perform high-order abstract modeling of user and item attribute text, the problems of feature distribution shift and overfitting can be effectively mitigated. Through multi-layer nonlinear transformations, the expressive power of semantic embedding is enhanced, providing fundamental support for cold-start and heterogeneous attribute recommendation, significantly outperforming recommendation methods that only use collaborative information. S400. Construct a user-item bipartite graph based on user-item interaction data, and introduce residual connection mechanism, node attention mechanism and robust enhancement mechanism to generate collaborative embedding representation.

[0043] In this step, a user-item bipartite graph is constructed based on user-item interaction data. Specifically, based on the user-item interaction data, the user ID is first mapped to... Item number mapped to and each interaction pair Convert to undirected edge and Construct a sparse adjacency matrix, represented as: ; To address the issues of sparse graph structures, significant node noise, and oversmoothing during deep propagation, this embodiment introduces residual connection mechanisms, node attention mechanisms, and robustness enhancement mechanisms. These mechanisms improve the modeling capability of high-order cooperative relationships, enhance the effectiveness of information propagation paths, and overcome the oversmoothing and noise sensitivity problems present in traditional GCN structures. They also improve deep robustness while maintaining shallow-level discrimination capabilities.

[0044] In this step, the residual connection mechanism includes two parts: graph convolution propagation and residual fusion. Residual connections are added to the graph convolutional network, and the input of the previous layer and the output of the current layer are fused proportionally and weighted. The specific process is as follows: (1) Graph convolution propagation: In the graph structure Layer representation of the input nodes of this layer Perform standard graph convolution calculations to obtain a temporary representation of the graph after convolution: ; in: Normalized adjacency matrix (commonly used) ); For the first Layer node representation; This is the learnable weight matrix for this layer; This represents the temporary representation of the graph after convolution.

[0045] (2) Residual fusion: To avoid information loss and over-smoothing, the input nodes of this layer are represented as follows: Temporary representation after convolution with graph By performing weighted fusion, the final result is obtained. Layered embedding representation: ; in, The residual weighting coefficient can be set to a fixed value (e.g., ) or learnable parameters; when When the value approaches 1, more emphasis is placed on the convolutional information of the current layer; when... When the value approaches 0, the original features of the input layer are preserved more; elements are linearly fused along the dimension, effectively improving the multi-path propagation capability of information.

[0046] Node-level attention is used to assign attention weights to neighboring nodes. It aggregates features based on the importance of neighbors to the target node, enabling the model to automatically learn the importance of different neighbors and dynamically allocate aggregated weights, thereby improving the expressive power and robustness of structural modeling. The specific process is as follows: For any pair of nodes in the user-item bipartite graph ,in express To find neighboring nodes, first calculate the neighboring nodes. For the target node Attention coefficient This indicates its contribution weight. First, the target node... Its neighboring nodes After feature concatenation, linear mapping and nonlinear transformation are performed, and then the attention scores of all neighboring nodes are normalized by softmax operation to obtain the importance distribution of neighboring nodes. This process can be represented as: ; in, and Representing the target node respectively and its neighboring nodes Input features, It is a learnable parameter matrix used for feature transformation. This represents a vector concatenation operation. It is a learnable attention vector.

[0047] Subsequently, the target node The representation vector in the th The layer obtains the collaborative embedding representation of the target node in the current layer by weighted aggregation of the feature vectors of its neighboring nodes. This process can be represented as: ; Each neighbor feature in the formula is assigned an attention score. Weighted summaries are used to obtain the target node. In the current layer (i.e., the first) Collaborative embedding representation of layers .

[0048] Robust enhancement mechanisms include edge dropout operations and embedding perturbation strategies.

[0049] Specifically, the edge sampling operation involves randomly retaining or deleting edges in the adjacency matrix of the user-item bipartite graph before each layer of graph convolution propagation, based on probability. Assuming a retention rate of... That is, each edge has The probability is retained for propagation. The embedding perturbation strategy injects Gaussian noise into the node representation during embedding propagation, expressed as: ; in: Represents a node In the Representation vectors in layered graph neural networks; The injected noise term has a mean of 0 and a covariance of... Multidimensional Gaussian distribution, It is an identity matrix, ensuring that the noise is independent and identically distributed in all dimensions; `noise intensity` is a hyperparameter used to control the magnitude of the perturbation. The formula means that a random noise vector following a Gaussian distribution is added to the current node embedding vector. The aim is to make the model more tolerant of input perturbations during training, thereby learning more stable and better generalizing node representations. Especially when no perturbation is used during testing, the model performs more robustly on "clean inputs," demonstrating the robustness improvement strategy of "adding perturbation during training, making it more robust during testing."

[0050] S500 aligns collaborative embedding representations and high-order semantic embedding representations based on a multi-head self-attention mechanism, and uses a gating mechanism for weighted fusion. Based on a self-supervised learning strategy, it combines distillation loss and cross-modal contrastive loss to constrain the fusion embedding and generate the final fused embedding representation.

[0051] To better embed collaborative representation and the high-order semantic embedding representation generated through deep semantic encoding To achieve effective integration, this step introduces a two-level fusion mechanism, which aligns collaborative embedding representations and high-order semantic embedding representations based on a multi-head self-attention mechanism, and uses a gating mechanism for weighted fusion.

[0052] Multi-head attention constructs three sets of projections—query, key, and value—to calculate attention weights and generate a fused representation. This is achieved through collaborative embedding. and higher-order semantic embedding representation For input, its attention process is as follows: ; in: Represents the query vector, derived from collaborative embedding. Obtained through linear transformation; and These are the key and value, derived from semantic embedding. ; This is the corresponding learnable parameter matrix; all vectors have a dimension of 1. .

[0053] ; in: This indicates scaled dot product similarity, preventing the gradient vanishing problem in high-dimensional spaces; Used to normalize attention weights, emphasizing semantic features most relevant to co-embeddings; weighted... This represents the feature representation after fusion.

[0054] To enhance the model's ability to model across multiple semantic spaces, H attention heads are introduced for parallel computation. Each head uses an independent projection matrix to perform the aforementioned attention process, as shown below: ; The fused feature representation of each head output After being concatenated, the data is uniformly mapped back to the original dimensions to obtain attention-fused features: ; in, To output the mapping matrix, This indicates a vector concatenation operation.

[0055] Multi-head attention mechanisms enable independent modeling of different semantic dimensions through projection and aggregation of multiple parallel subspaces, and improve the model's ability to model heterogeneous modal dependencies.

[0056] To achieve adaptive fusion between collaborative embedding representations and semantic embeddings (such as attention-fusion features), a gated fusion mechanism is introduced. This mechanism dynamically adjusts the importance of different information sources based on the features of each sample, thereby improving the modeling ability for diverse user preferences and heterogeneous item attributes. Given attention-fusion features... With collaborative embedding representation The fusion process is as follows: First, and The concatenation is performed, and the input is fed into a fully connected network. The gating vector is obtained through the sigmoid activation function. : ; in: This indicates vector concatenation; and These are the weight matrix and bias term of the gated network, respectively; For the Sigmoid function, the gate value is... ;vector Each dimension represents the proportion of collaboration and semantic information fused in that dimension.

[0057] Subsequently, based on the gate vector Attention fusion features With collaborative embedding representation Weighted fusion is performed to obtain the final multimodal fused embedding representation. : ; in: Represents element-wise multiplication (Hadamard Product); gated vector Attention fusion features for controlling semantics With collaborative embedding representation The fusion ratio.

[0058] The cross-modal fusion mechanism proposed in this step utilizes multi-head attention to align the semantic dimensions of collaborative embeddings and semantic embeddings, and then dynamically adjusts the fusion weight through a gating network to achieve adaptive integration of the two modalities. Compared with fixed fusion strategies, this method has stronger representational flexibility and personalized modeling capabilities, and can automatically adjust the fusion strategy according to the data distribution.

[0059] To further integrate collaborative and semantic information and improve representation consistency and generalization ability, this embodiment introduces dynamic knowledge distillation and diversified contrastive learning on the basis of the main supervision objective (such as BPR loss) to construct alignment constraints and discriminative enhancement that integrate the collaborative and semantic sides, as detailed below: (1) Dynamic Knowledge Distillation The fusion representation of collaborative embedding and semantic embedding is denoted as the multimodal fusion embedding representation. The high-order semantic embedding representation generated after deep semantic encoding is denoted as The similarity between the two is calculated using cosine similarity, and the specific formula is as follows: ; in, This represents the dot product of two vectors; and Represent the two vectors respectively Norm (i.e., modulus); A larger value indicates that the collaborative representation is more similar to the semantic representation.

[0060] To encourage consistency between collaboration and semantic representation, a distillation loss term is introduced: ; in, The squared Euclidean distance between the multimodal fusion embedding representation and the high-order semantic embedding representation measures the degree of inconsistency between the two. The weight of the distillation loss determines the importance of this loss term in the total loss. It is dynamically adjusted with similarity to reflect the degree of consistency between the two embeddings, and can be expressed as: ; in: The basic weight for distillation; The adjustment coefficient controls the effect of similarity on distillation intensity; when A higher value indicates greater consistency between the coherence and semantics; in this case, increasing the distillation intensity further aligns the results. The lower the value, the smaller the distillation weight automatically becomes, avoiding imposing excessive convergence constraints on unaligned representations and improving training stability.

[0061] (2) Contrastive Learning In this step, the diversified contrastive learning includes original embedding contrast and fused embedding contrast. The original embedding contrast is applied to the co-embedded representation. With the original semantic embedding To improve the alignment between the original modes, a consistency constraint is imposed, and its contrast loss is defined as: ; in, For collaborative embedding representation, and For semantic embedding, Positive sample pairs are those with collaborative semantics, meaning they originate from the same user or item; exist When the time is a negative sample pair, that is, a combination of collaboration and semantic mismatch; Represents the cosine similarity function; The temperature coefficient controls the sensitivity of the comparison similarity distribution; a smaller value will amplify the similarity differences. Further introduction of multimodal fusion embedding representation With semantic embedding A consistency-constrained fusion embedding comparison is performed to ensure that the fusion representation maintains a high degree of consistency with the original semantic embedding while integrating collaborative and semantic information. The comparison loss is defined as: ; in, For multimodal fusion embedding representation, and This is semantic embedding.

[0062] Combining the two branches above, the final cross-modal contrast loss is defined as: ; This cross-modal contrastive learning objective not only effectively improves the consistency between representations of different modalities, but also enhances the discriminativeness of representations through the guidance of negative samples, thereby improving the model's ability to generalize semantic information while optimizing recommendation performance.

[0063] The final total loss function integrates the objectives of master supervision and self-supervision as follows: ; in, Loss due to self-supervised learning; Enhanced semantic-co-alignment; Strengthen the discriminative power between semantics and collaboration; and The balancing coefficient for the loss term can be dynamically adjusted using the validation set, or its weight can be increased later in the training process to enhance embedding consistency.

[0064] This step introduces a self-supervised optimization mechanism, enhancing the consistency and discriminative ability between collaborative and semantic modalities through dynamic knowledge distillation and diversified contrastive learning. The distillation process employs a similarity-based dynamic weight adjustment strategy, allowing the fused embedding to gradually approach the representation space of the semantic embedding, thus improving semantic alignment. Simultaneously, the contrastive loss design strengthens the contrastive constraints between modalities, enhancing the model's generalization ability and reducing the impact of modality bias.

[0065] S600. Output recommendation results based on the final fused embedded representation.

[0066] In this step, after obtaining the final fused embedding representation, the recommendation generation process based on dot product scoring and mask sorting generates and outputs the recommendation results.

[0067] First, full embedding extraction is performed. The fused embedding matrix is ​​obtained by calling the forward propagation function, and is represented as follows: ; in, The merged user and item integration matrix is ​​used for embedding. Total number of users; The total number of items; For the embedded vector dimension; For the first A fused embedded representation of an entity (user or item).

[0068] Then, click-based scoring is performed on a group of users to be predicted. Extract its corresponding embedding submatrix, represented as:

[0069] Embed it with all items Performing a matrix dot product yields a fractional matrix, represented as:

[0070] in, This represents the number of users predicted in this batch. Embedded matrix for batch users; Merge and embed the matrix for all items. The transpose of the embedding matrix is ​​used to fuse all items; For the scoring matrix, the first... Elements for users For items The predicted score.

[0071] Subsequently, training masking is performed, utilizing the training set interactive mask matrix. To prevent repeated recommendations of items that have already been interacted with, the system will hide items that have already been interacted with. Specifically, this will be done as follows:

[0072] in Interaction mask for the training set; Indicates user Interacted items ; This is the scoring matrix after masking.

[0073] Subsequently, based on the threshold Keep all The items are used to create a recommendation list; alternatively, a Top-K sort is performed, and each row is sorted... Take the first result in descending order. The item index with the maximum value is used to form a recommendation list. This process is represented as follows:

[0074] in, For users generated Recommended item index set.

[0075] Finally, output the recommendation list for each user. It is used for downstream recommendation tasks.

[0076] Example 2 In a typical embodiment of the present invention, this embodiment discloses a recommendation system based on cross-modal collaborative-semantic alignment, including: The data acquisition module is configured to: acquire text attribute data and user-item interaction data from user information and item information; The semantic embedding acquisition module is configured to acquire user semantic embedding and item semantic embedding based on user information text attributes and item information text attributes. The high-order semantic embedding generation module is configured to perform multi-layer perceptual deep semantic encoding on user semantic embedding and item semantic embedding respectively to generate high-order semantic embedding representations; The collaborative embedding generation module is configured to: construct a user-item bipartite graph based on user-item interaction data, introduce residual connection mechanism, node attention mechanism and robust enhancement mechanism, and generate collaborative embedding representation; The fusion embedding generation module is configured to: align collaborative embedding representations and high-order semantic embedding representations based on a multi-head self-attention mechanism, perform weighted fusion using a gating mechanism, and constrain the fusion embedding based on a self-supervised learning strategy, using joint distillation loss and cross-modal contrastive loss to generate the final fusion embedding representation; The recommendation module is configured to output recommendation results based on the final fused embedding representation.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A recommendation method based on cross-modal collaborative-semantic alignment, characterized in that, include: Retrieve text attribute data and user-item interaction data from user and item information; Based on the text attributes of user information and the text attributes of item information, obtain the semantic embedding of users and the semantic embedding of items; Multi-layer perceptual deep semantic encoding is performed on user semantic embeddings and item semantic embeddings respectively to generate high-order semantic embedding representations; Based on user-item interaction data, a user-item bipartite graph is constructed, and residual connection mechanism, node attention mechanism and robust enhancement mechanism are introduced to generate collaborative embedding representation; Aligning collaborative embedding representations and high-order semantic embedding representations based on multi-head self-attention mechanism, and using gating mechanism for weighted fusion, the final fused embedding representation is generated by combining distillation loss and cross-modal contrastive loss based on self-supervised learning strategy. The recommendation results are output based on the final fused embedding representation.

2. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 1, characterized in that, The deep semantic encoding process is as follows: in, For user semantic embedding vectors or item semantic embedding vectors; For the first Layer weights and bias coefficients, This is a high-order semantic embedding representation generated through deep semantic encoding.

3. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 1, characterized in that, The residual connection mechanism is as follows: In the graph structure Each layer performs standard graph convolution on its input node representations to obtain a temporary representation after graph convolution; the input node representations of this layer are then residually fused with the temporary representation after graph convolution to obtain the final representation. Layered embedding representation.

4. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 1, characterized in that, The node attention mechanism is used to assign attention weights to neighboring nodes and aggregate features based on the importance of neighbors to the target node. Specifically, the features of the target node and its neighboring nodes are concatenated and then linearly mapped and nonlinearly transformed. The attention scores of all neighboring nodes are normalized by softmax operation to obtain the importance distribution coefficients of the neighboring nodes. The target node's representation vector is then weighted and aggregated with the feature vectors of its neighboring nodes in the next layer to obtain the embedding representation of the target node in the current layer.

5. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 1, characterized in that, The robust enhancement mechanism includes edge sampling operation and embedding perturbation strategy; the edge sampling operation is to randomly retain or delete adjacent edges in the user-item bipartite graph according to probability; the embedding perturbation strategy is to inject Gaussian noise into the node representation during the embedding propagation process.

6. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 1, characterized in that, The multi-head self-attention mechanism aligns the collaborative embedding representation and the higher-order semantic embedding representation. Specifically, the collaborative embedding representation and the higher-order semantic embedding representation are used as inputs. Multiple attention heads calculate the attention weights of each head in parallel and generate a fused feature representation. The fused feature representations generated by each head are concatenated and mapped back to the original dimension to obtain the attention fusion feature.

7. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 6, characterized in that, Weighted fusion is performed using a gating mechanism, specifically: attention fusion features and collaborative embedding representations are concatenated and input into a fully connected layer, and a gating vector is obtained through a sigmoid activation function; based on the gating vector, attention fusion features and collaborative embedding representations are weighted and fused to obtain a multimodal fusion embedding representation.

8. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 7, characterized in that, Distillation loss is specifically expressed as follows: ； in, The squared Euclidean distance between the multimodal fusion embedding representation and the high-order semantic embedding representation measures the degree of inconsistency between the two. The weights for the distillation loss are dynamically adjusted based on the cosine similarity between the multimodal fusion embedding representation and the higher-order semantic embedding representation, and are expressed as: ； in, The basic weight for distillation; This is the adjustment coefficient; Cosine similarity between the multimodal fusion embedding representation and the higher-order semantic embedding representation.

9. The recommendation method based on cross-modal collaborative-semantic alignment as described in claim 7, characterized in that, Cross-modal contrastive loss includes original embedding contrastive loss and fused embedding contrastive loss; specifically, the original embedding contrastive loss is as follows: ； in, For collaborative embedding representation, and For semantic embedding, Positive sample pairs are those with collaborative semantics, meaning they originate from the same user or item; exist When the time is a negative sample pair, that is, a combination of collaboration and semantic mismatch; Represents the cosine similarity function; The temperature coefficient controls the sensitivity of the comparison similarity distribution; a smaller value will amplify the similarity differences. The specific fusion embedding contrast loss is as follows: ； in, For multimodal fusion embedding representation, and For semantic embedding; The cross-modal contrast loss is as follows: ； The total loss function is specifically expressed as follows: ； in, Loss due to self-supervised learning; Enhanced semantic-co-alignment; Strengthen the discriminative power between semantics and collaboration; and The balancing coefficient for the loss term is dynamically adjusted using the validation set.

10. A recommendation system based on cross-modal collaborative-semantic alignment, characterized in that, include: The data acquisition module is configured to: acquire text attribute data and user-item interaction data from user information and item information; The semantic embedding acquisition module is configured to acquire user semantic embedding and item semantic embedding based on user information text attributes and item information text attributes. The high-order semantic embedding generation module is configured to perform multi-layer perceptual deep semantic encoding on user semantic embedding and item semantic embedding respectively to generate high-order semantic embedding representations; The collaborative embedding generation module is configured to: construct a user-item bipartite graph based on user-item interaction data, introduce residual connection mechanism, node attention mechanism and robust enhancement mechanism, and generate collaborative embedding representation; The fusion embedding generation module is configured to: align collaborative embedding representations and high-order semantic embedding representations based on a multi-head self-attention mechanism, perform weighted fusion using a gating mechanism, and constrain the fusion embedding based on a self-supervised learning strategy, using joint distillation loss and cross-modal contrastive loss to generate the final fusion embedding representation; The recommendation module is configured to output recommendation results based on the final fused embedding representation.

Images