Multimodal graph perception item recommendation method and system based on large model semantic guidance

Abstract

The disclosure provides a multi-modal graph perception item recommendation method and system based on large model semantic guidance, relates to the technical field of multi-modal recommendation, and comprises the following steps: performing global semantic analysis on an item category system, inferring a semantic community description text, vectorizing and encoding the semantic community description text, and generating shared semantic initialization anchor points of hypergraph visualization and text; performing individualized importance measurement on the interaction edges of an interaction graph, constructing an inverse tendency main graph, and obtaining local individualized representation by using graph convolution; constructing a cross-modal item similarity graph, and extracting latent semantic representation between items; performing hypergraph convolution on the prototype hyperedge based on the shared semantic initialization anchor points, and obtaining community-level global semantic representation; and performing multi-level representation fusion and outputting preference prediction scores by using cross-modal alignment loss and a joint optimization objective function to fuse the local individualized representation, the latent semantic representation between items, and the community-level global semantic representation. The disclosure improves the accuracy of recommendation.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal recommendation technology, specifically to a multimodal graph-aware item recommendation method and system based on large model semantic guidance. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the continuous expansion of the scale of internet content, users' preferences are often deeply influenced by the multimodal content carried by items, such as product images, text descriptions, and category tags. This has prompted researchers to integrate multimodal features with collaborative filtering signals, which has promoted the rise and prosperity of multimodal recommendation as a research direction.

[0004] Graph Neural Networks (GNNs) have become the mainstream technical paradigm in the field of multimodal recommendation due to their natural ability to model high-order connectivity in user-item bipartite graphs. However, existing multimodal recommendation methods still have the following limitations: (1) The homogenization problem of representation caused by full graph propagation. Current mainstream methods generally perform message propagation on the complete user-item interaction graph. As the number of propagation layers increases, node embeddings gradually converge to the global mean. High-frequency individualized signals reflecting unique user preferences are continuously smoothed, eventually leading to similar representations among different users and a degradation in personalized modeling capabilities. At the same time, real interaction data inevitably contains noise behaviors caused by accidental clicks, occasional browsing, etc. These low-quality interaction edges are treated equally and repeatedly spread during full graph propagation, further interfering with the quality of node representations.

[0005] (2) The problem of missing global dependencies due to insufficient local receptive field. Local graph propagation based on a finite number of hops has inherent limitations in receptive field: user groups with highly similar preferences are often distributed in different connected regions that are far apart in the interaction graph, and it is difficult to capture their potential common interests by neighborhood aggregation with a finite number of hops alone; for users with sparse historical interactions, the amount of information provided by their local neighborhood itself is limited, and the quality of the learned user representation is difficult to guarantee without global semantic supplementation. The hypergraph encoders of visual and textual modalities are independent of each other and lack explicit cross-modal semantic constraints, which may introduce semantic conflict noise during fusion.

[0006] (3) The problem of missing semantic anchoring of hypergraph prototype vectors. In existing hypergraph recommendation methods, the prototype vectors corresponding to hyperedges usually adopt a random initialization strategy, relying entirely on subsequent gradient optimization to gradually converge to meaningful semantic cluster centers. This mechanism has two hidden dangers: on the one hand, the soft assignment of prototype vectors to items in the random initialization state is almost uniformly random, lacking effective early gradient signals, resulting in slow model convergence speed; on the other hand, the fully data-driven optimization process lacks prior semantic constraints, and prototype vectors are at risk of converging to latent space regions that are difficult for humans to interpret, thereby affecting the semantic rationality of item-hyperedge assignment and the optimization effect of cross-modal global alignment loss. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes a multimodal graph-aware item recommendation method and system based on large-model semantic guidance. It performs two-stage hypergraph convolutional propagation based on prototype hyperedges initialized with semantic guidance to capture community-level global dependencies across local neighborhoods. By constraining the semantic consistency of visual and textual global representations through cross-modal contrastive alignment loss, it weights and fuses multi-level representations and outputs preference prediction scores, achieving collaborative convergence of visual and textual global hypergraph representations in the semantic space.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: Multimodal graph-aware item recommendation methods based on large-model semantic guidance include: Obtain the item category system and construct a user-item interaction graph; Using a large model, a global semantic analysis of the item category system is performed to infer the semantic community description text. The semantic community description text is then vectorized and encoded to generate shared semantic initialization anchor points for both the hypergraph visual and textual prototype matrices. Individual importance measures are performed on the interaction edges of the user-item interaction graph, an inverse tendency main graph is constructed, and graph convolution mean pooling is used to aggregate the graph to obtain local individual representations; a cross-modal item similarity graph is constructed to extract latent semantic representations between items; Based on shared semantic initialization anchor points, hypergraph convolution is performed on prototype hyperedges to obtain community-level global semantic representations; By using cross-modal alignment loss and joint optimization objective function, multi-level representation fusion of local individualized representation, inter-item latent semantic representation and community-level global semantic representation is performed to output preference prediction scores and a recommendation list.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A multimodal graph-aware item recommendation system based on large-model semantic guidance includes: The semantic community guidance module is used to obtain the item category system and construct the user-item interaction graph; it uses a large model to perform global semantic analysis on the item category system, infers the semantic community description text, vectorizes the semantic community description text, and generates shared semantic initialization anchor points for the hypergraph visual and text prototype matrices. The main graph denoising module is used to measure the individual importance of the interaction edges of the user-item interaction graph, construct an inverse tendency main graph, and aggregate it using graph convolution mean pooling to obtain a local individualized representation; The Latent Semantic Graph module is used to construct a cross-modal item similarity graph and extract latent semantic representations between items; The global hypergraph encoding module is used to perform hypergraph convolution on the prototype hyperedges based on shared semantic initialization anchors to obtain a community-level global semantic representation; The feature fusion and prediction module is used to fuse local individualized representations, inter-item latent semantic representations, and community-level global semantic representations through cross-modal alignment loss and joint optimization objective function, and output preference prediction scores and recommendation lists.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program that, when executed by a processor, implements the aforementioned multimodal graph-aware item recommendation method based on large model semantic guidance.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the multimodal graph-aware project recommendation method based on large model semantic guidance.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the multimodal graph-aware project recommendation method based on large model semantic guidance.

[0013] Compared with the prior art, the beneficial effects of this disclosure are as follows: This disclosed multimodal graph-aware item recommendation method, guided by the semantics of a large model, introduces the prior semantic knowledge of the large model into the initialization process of the hypergraph prototype vector, overcoming the limitations of existing methods that rely on random initialization, leading to semantic confusion in early assignment and slow convergence. The introduction of semantic anchors ensures that the prototype vectors have a reasonable semantic distribution from the beginning of training, accelerating the effective convergence of the hypergraph structure and providing a consistent semantic benchmark for optimizing the cross-modal global alignment loss.

[0014] This disclosed multimodal graph-aware item recommendation method based on large model semantic guidance adaptively constructs a denoised sparse main graph in each training round through an inverse bias probability sampling mechanism. This effectively suppresses the representation homogenization problem caused by full graph propagation, while retaining the individualized interaction edges that are most discriminative for personalized recommendations, thus significantly improving the quality of local collaborative representations.

[0015] This disclosed multimodal graph-aware project recommendation method based on large model semantic guidance uses the semantic community description generated by the large model as a cross-modal shared initialization benchmark. Combined with the continuous constraint of the InfoNCE cross-modal contrastive alignment loss, it achieves the collaborative convergence of visual and textual global hypergraph representations in the semantic space, fundamentally alleviating the cross-modal semantic bias problem introduced by independent hypergraph encoding and improving the fusion quality of global embedding. Attached Figure Description

[0016] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0017] Figure 1 This is a flowchart of a multimodal graph-aware project recommendation method based on large model semantic guidance, according to an embodiment of this disclosure. Figure 2 This is a schematic diagram of the structure of a multimodal graph-aware item recommendation system based on large model semantic guidance, according to an embodiment of this disclosure. Detailed Implementation

[0018] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0021] Example 1 One embodiment of this disclosure provides a multimodal graph-aware item recommendation method based on large model semantic guidance, the method steps of which include: Step 1: Obtain the item category system and construct the user-item interaction graph; use the large model to perform global semantic analysis on the item category system, infer the semantic community description text, vectorize the semantic community description text, and generate shared semantic initialization anchor points for the hypergraph visual and text prototype matrices. Step 2: Perform individualized importance measurement on the interaction edges of the user-item interaction graph, construct an inverse tendency main graph, and aggregate it using graph convolution mean pooling to obtain local individualized representations; Step 3: Construct a cross-modal item similarity graph and extract latent semantic representations between items; Step 4: Perform hypergraph convolution on the prototype hyperedge based on the shared semantic initialization anchor points to obtain the community-level global semantic representation; Step 5: By using cross-modal alignment loss and joint optimization objective function, multi-level representation fusion of local individualized representation, inter-item latent semantic representation and community-level global semantic representation is performed, and preference prediction score is output, resulting in a recommendation list.

[0022] As one embodiment, this disclosure presents a multimodal graph-aware item recommendation method based on large model semantic guidance, which is a main graph denoising and global hypergraph recommendation method (LLM-PHGRec) guided by large model semantics. This method starts with large model semantic community guidance, first performing global semantic analysis on the item category system of the dataset, inferring and generating K community descriptions with distinct semantic discriminative power, and using their encoding results as semantic initialization anchors for the hypergraph prototype vector, providing a meaningful prior structure for subsequent learning of the global hypergraph; subsequently, it uses inverse tendency probability sampling based on normalized Laplacian edge weights. The mechanism dynamically constructs a denoised sparse main graph before each training round, prioritizing the retention of individualized preference edges connecting low-degree nodes while eliminating low-confidence noise interactions. Next, collaborative signals and semantic associations are extracted from the dual-path local modeling structure formed by the sparse main graph and the item-item latent semantic affinity graph. Then, based on semantically guided initialization of prototype hyperedges, a two-stage hypergraph convolutional propagation is performed to capture community-level global dependencies across local neighborhoods. Finally, cross-modal contrastive alignment loss constrains the semantic consistency of visual and textual global representations, weightedly fusing multi-level representations and driving model training with a joint optimization objective.

[0023] Problem definition.

[0024] Let the user set be The collection of items is User-item history interaction records are generated by an interaction matrix. express, and These represent the number of users and the number of items, where if the number of users Previously with items When interaction occurs Otherwise, it is 0. Interaction data naturally forms a user-item bipartite graph. , where the node set edge set ; and The diagrams are shown below. Chinese users With items The set of neighboring nodes, and This corresponds to the degree.

[0025] Each item Associate the original multimodal features extracted by the pre-trained encoder. Define the modality set. ,in Representing visual modality, Represents a text modality. Items In modality The original feature vector is denoted as ,in This represents the original feature dimension of the modality; all items in the modality and The characteristic matrices below are denoted as follows: and .

[0026] Given an interaction matrix With multimodal features { The goal of the model is to learn a scoring function. For users With items The probability of interaction between them is predicted, where This represents all learnable parameters. Ultimately, a Top-K recommendation list is generated for each user.

[0027] The specific implementation process of the method proposed in this disclosure to solve the above problems is as follows.

[0028] Step 1: Obtain the item category system and construct the user-item interaction graph; use the large model to perform global semantic analysis on the item category system, infer the semantic community description text, vectorize the semantic community description text, and generate shared semantic initialization anchor points for the hypergraph visual and text prototype matrices.

[0029] In hypergraph recommendation methods, the prototype vectors corresponding to the K hyperedges are the core parameters for modeling the semantic community affiliation of items. If a random initialization strategy is adopted, the soft assignment of items to prototype vectors in the early stages of training is almost uniformly random, lacking effective gradient signal guidance, resulting in a slow and unstable semantic community formation process. Simultaneously, the fully data-driven optimization path risks converging into a semantically ambiguous space, limiting the effectiveness of cross-modal global alignment. This disclosure introduces a large model to perform global semantic analysis of the item category system of the dataset offline before model training, providing initialization anchors with prior semantic basis for the prototype vectors. The specific steps are as follows: Step 11: Semantic community reasoning of the item category system; For a given dataset, the item information is derived from publicly available datasets on the Amazon e-commerce website. The information is obtained from the items themselves, namely the images and text descriptions associated with the items.

[0030] First, its item collection By statistically summarizing the category tags of all items, a category distribution set at the dataset level is obtained:

[0031] in, This represents the total number of different target tags in the dataset. For the first Individual target signature text. With For input, construct a prompt template for semantic community induction. Guide the large model to perform the following inference tasks:

[0032] in, This is the preset number of hyperedges, i.e., the number of semantic communities; Output for large models A collection of semantic community description texts. No. Natural language description of a semantic community.

[0033] Prompt Template The large model is explicitly required to perform the following reasoning tasks: first, understand the overall picture of the item domains covered by the dataset; second, based on the semantic relevance between categories... Categories were grouped together Each semantic community possesses inherent consistency and distinctness. Finally, a natural language description balancing generalization and discriminability is generated for each community. Taking the Amazon Baby dataset as an example, when... At that time, a large model might generate the following typical community description: Feeding and complementary food equipment for infants (bottles, breast pumps, etc.); For infant sleep and comfort products (cribs, pacifiers, etc.); For infant travel and safety (strollers, car seats, etc.); For infant bathing and daily care (bathtubs, skincare products, etc.). The above semantic community description accurately reflects the functional distribution structure of items in this domain and has clear semantic distinguishability.

[0034] As one example, the entire semantic reasoning process only calls the large model once, generating... K The community description text remains constant throughout the entire training process, without introducing any additional inference overhead during training, demonstrating good engineering feasibility.

[0035] Step 12: Vectorized encoding of semantic community description text; Generate large models The semantic community description text is encoded using a pre-trained text encoder (such as Sentence-BERT) consistent with the feature extraction of item text, resulting in... Semantic community vectors:

[0036] in For the first Dense vector representation of semantic communities, This is the output dimension of the text encoder. The community vectors are concatenated column-wise to form a semantic community matrix:

[0037] Each column corresponds to a community center vector obtained from semantic reasoning of the large model, which encodes the core features of the community at the semantic level.

[0038] Step 13: Semantic anchoring initialization of the hypergraph prototype matrix; This embodiment describes each mode. Maintain a learnable prototype matrix Each column Corresponding to the hypergraph A superedge represents a mode. Next The prototype vector of a semantic community. v , t They represent the visual modality and the text modality, respectively.

[0039] For the text modality prototype matrix Directly using the semantic community matrix As initial values. For the visual modality prototype matrix. The following two-stage initialization strategy is adopted. First, the semantic community matrix is ​​initialized. Apply a linear projection to map the text semantic vectors to the visual feature space:

[0040] in For cross-modal projection matrix, These are bias vectors, all initialized using a uniform Xavier distribution. The projection results are then subjected to L2 column-by-column normalization.

[0041] Through the aforementioned initialization strategy, the visual and textual prototype matrices are jointly anchored to the semantic community structure obtained from the large model's inference from the very beginning of training, possessing an initial state of cross-modal semantic consistency. This design brings two key advantages: first, the soft assignment of items to hyperedges in the early stages of training is semantically reasonable, accelerating the effective convergence of the hypergraph structure; second, it provides a consistent semantic benchmark for the subsequent optimization of cross-modal global alignment loss, enabling the alignment of visual and textual global representations to proceed in a more meaningful semantic direction.

[0042] and All models start from the aforementioned semantic anchoring state and are updated along with all model parameters via gradient descent during subsequent training. The large model is only responsible for providing the initial anchor point of semantic prior and does not participate in any training-time calculations.

[0043] Step 2: Perform individualized importance measurement on the interaction edges of the user-item interaction graph, construct an inverse tendency master graph, and aggregate it using graph convolution mean pooling to obtain a local individualized representation.

[0044] Existing multimodal recommendation methods perform graph convolutional propagation on the complete user-item interaction graph, essentially applying low-pass filtering to the graph signal. As the propagation layer increases, node representations gradually converge to the global mean, continuously diluting the individualized signals reflecting unique user preferences. Simultaneously, noise edges embedded in the real interaction data are amplified during full graph propagation, further degrading the quality of node representations. This embodiment proposes an inverse preference probability sampling mechanism based on normalized edge weights, dynamically constructing a denoised sparse main graph before each training round, eliminating noise interference while enhancing the preservation of individualized preference edges. The specific steps are as follows: Step 21: Individualize the importance of interactive edges; User-item interaction graph Each edge in Individual importance weights are assigned based on the degree of the nodes they connect:

[0045] in, and users respectively With items In the interaction diagram The weighting is consistent with the symmetric normalization term of the normalized graph Laplacian matrix, providing a clear semantic interpretation: the interaction edges connecting low-degree users and low-degree items have higher weights, and such interactions are more likely to stem from the user's genuine individual preferences rather than herd behavior; the edge weights connecting high-degree nodes are relatively low, and the information they carry is mainly global collaborative signals with weaker individual differentiation capabilities.

[0046] Step 22: Dynamically construct the reverse trend master graph; Before each round of training, the set of interaction edges is sampled using individualized importance weights as sampling probabilities. Perform weighted sampling with replacement to construct the current round's sparse main graph. :

[0047] Let the sampling retention ratio be Then each round of the main chart contains Interactive edge, The edge set retained from the sampling is then re-symmetrically normalized to construct a normalized adjacency matrix for the sparse main graph:

[0048] in, Main image The adjacency matrix, This is the corresponding degree matrix. Main image. Dynamic reconstruction before each training round and the injection of randomness allow the model to observe diverse denoised structural views in different training rounds, which has the effect of implicit data augmentation and enhances the model's robustness to interactive noise.

[0049] Step 23: Construct the initial embedding; For each mode The original feature vector of the item is transformed through linear transformation. Projected to a unified embedding dimension :

[0050] in, For learnable projection matrices, The bias vector is projected, and the features are L2 normalized to ensure numerical stability. For each user... Maintain two independent learnable embedding vectors, one for vision and one for text. Initialized with a uniform Xavier distribution. Two embedding paths are concatenated to form the initial embeddings of the user and the item:

[0051] in, This represents a vector concatenation operation. .

[0052] Step 24: Perform lightweight graph convolution propagation on the main graph; The initial embedding matrix of all users and items Input a sparse main graph and execute... Layered LightGCN-style graph convolutional propagation:

[0053] through After layer propagation, the outputs of each layer are aggregated by mean pooling to mitigate mutual interference between neighborhood information with different hop counts.

[0054] from Extract user main image representation from each. Representation of the main image of the item The main image representation of a single user and item is denoted as follows: and .

[0055] The training phase uses a sparse main graph that is dynamically reconstructed round by round. To inject structural diversity; the reasoning phase switches to a complete interactive graph. The full normalized Laplace matrix is ​​used to ensure that no interaction information is discarded during testing.

[0056] Step 3: Construct a cross-modal item similarity graph to extract latent semantic representations between items.

[0057] Main graph modeling extracts individualized signals at the user-item interaction level from a collaborative filtering perspective, but its information source is limited to explicit behavioral records. In actual recommendation scenarios, implicit similarities between items in visual style, functional attributes, or textual semantics are also important clues revealing potential user preferences—two items that have never been interacted with by the same user, if highly similar in modal feature space, often have a strong overlap in the preference distribution of their potential audiences. This embodiment constructs a KNN-based item-item semantic affinity graph in the visual and textual modal feature spaces to supplement the implicit semantic relationships between items that the interaction graph cannot capture. The specific steps are as follows: Step 31: Linear projection and normalization of modal features; After projecting the original feature vectors of the items, L2 row-wise normalization is performed to obtain the normalized feature matrices of the items under each modality:

[0058] in, For items In modality The normalized feature vectors are used to ensure the numerical stability of subsequent cosine similarity calculations.

[0059] Step 32: Construct a modality-specific KNN semantic affinity graph; For each mode Calculate the cosine similarity between all item pairs and construct a fully connected semantic similarity matrix. :

[0060] in, For items With items In modality The semantic similarity score is calculated below. Directly using a fully connected similarity matrix for graph propagation introduces a large number of weakly correlated noisy connections, and also brings... This reduces storage and computational overhead. Therefore, for each item node, only the one with the highest similarity is retained. The neighbors are used to obtain a sparse semantic adjacency matrix through KNN sparsification:

[0061] in, Represents items In modality The highest cosine similarity in the feature space A set of neighbors. Symmetric normalization is applied to the sparse adjacency matrix to eliminate the impact of node degree differences on feature propagation.

[0062] in, for The corresponding angle matrix.

[0063] Step 33: Weighted fusion of cross-modal semantic graphs; Visual and textual modalities characterize the semantic attributes of objects from complementary perspectives. A weighted linear fusion of the two normalized semantic graphs yields a latent semantic affinity graph adjacency matrix that integrates multimodal information.

[0064] in, The visual modality weight hyperparameter is used to balance the visual and textual semantic graphs. The latent semantic affinity graph is built during model initialization and remains fixed during training, resulting in relatively stable modal semantic relationships between items. The frozen graph structure can effectively reduce computational overhead during training.

[0065] Step 34: Feature propagation on the latent semantic graph; Item splicing and embedding Using the initial input, perform the operation on the latent semantic affinity graph. Layered graph convolution propagation:

[0066] Take the final layer output as the latent semantic graph representation of the item:

[0067] The latent semantic representation of a single item is denoted as . It encodes the implicit association information of items obtained through neighborhood aggregation in the multimodal semantic space.

[0068] Latent semantic graph propagation and main graph propagation are clearly complementary in terms of information source and modeling object: the main graph is based on user-item behavior interaction modeling, focusing on capturing individualized collaborative filtering signals; the latent semantic graph is based on item-item modal similarity modeling, focusing on mining cross-modal semantic associations that transcend interaction relationships. The two local modeling structures together constitute the complete foundation of local representation learning in this method.

[0069] Step 4: Perform hypergraph convolution on the prototype hyperedge based on the shared semantic initialization anchor point to obtain the community-level global semantic representation.

[0070] The dual-path local modeling structure constructed in steps 2 and 3 is constrained in its receptive field by the limited hop count neighborhood of the graph structure: user groups with highly similar preferences are often scattered across different connected regions of the interaction graph, making it difficult for the local propagation mechanism to effectively model their shared interests across local topologies; users with sparse historical interactions have limited information in their local neighborhoods, and their representation quality depends on the effective supplementation of global semantics. Hypergraphs, with their high-order expressive power of connecting any number of nodes with a single hyperedge, are naturally suitable for modeling group-level semantic dependencies. This step, based on the semantically guided prototype initialization completed in step 1, performs global hypergraph convolutional propagation driven by prototype hyperedges. The specific process is as follows: Step 41: Perform the Gumbel-Softmax soft assignment process from items to superedges; For each mode ,thing With the The correlation strength of a hyperedge is determined by the original features of the item. With prototype vector Inner product measure:

[0071] Items and The association scores of each hyperedge are organized as vectors. Directly to Softmax normalization forces items to be assigned to a single hyperedge, losing information about whether an item belongs to multiple semantic communities. This embodiment uses the Gumbel-Softmax mechanism to achieve differentiable soft assignment across multiple hyperedges:

[0072] in, For random noise sampled independently from the standard Gumbel distribution, This refers to temperature hyperparameters. When the output approaches 0, it tends to be hard-slotted (one-hot). When the value is large, the output tends to be uniformly distributed. Stacking the soft assignment vectors of all items row-wise yields the item-hyperedge incidence matrix:

[0073] in, The Line description of items In modality The distribution of membership degrees for each hyperedge in a hypergraph, the th The column reflects the first The distribution of items covered by a hyperedge represents the composition of the corresponding semantic community.

[0074] Compared to randomly initialized prototype vectors, semantically anchored initialization makes... The system can reflect the semantic affinity between items and various semantic communities in the early stages of training, thus making the early soft assignment results semantically reasonable and effectively avoiding gradient oscillations caused by random assignment in the early stages of training.

[0075] Step 42: Derive the user hyperedge association matrix; To propagate hyperedge semantic community information to the user side, sparse message propagation on the user side is performed using an interaction graph on the item-hyperedge association matrix, thus deriving the user-hyperedge association matrix:

[0076] in, For the complete normalized Laplace matrix of the user-item interaction graph, For the item-side association matrix Fill above The complete allocation matrix after the row zero matrix, Indicates before extraction The results corresponding to the user are as follows: This results in the final user-hyperedge association matrix. This operation aggregates the hyperedge affiliation distribution of a user's historical interactive items to the user side using normalized interaction weights, enabling the user's global representation to perceive the distribution characteristics of the semantic communities to which their historical interactive items belong. The user-side association matrix is ​​also processed using Gumbel-Softmax to maintain consistency with the item-side distribution.

[0077] Step 43: Two-stage hypergraph convolutional propagation; Represented by the main image of the item As initial input, execute Layer-two two-stage hypergraph convolutional propagation. layer The propagation process is divided into two stages: Phase 1: Item-to-hyperedge aggregation, which aggregates the features of items belonging to the same hyperedge into a community semantic representation for that hyperedge:

[0078] in , For the first Aggregated embedding of each hyperedge in a layer hypergraph convolution, the th The line represents the first The community representation is formed by a super-edge aggregating the items of its members.

[0079] Phase Two: Broadcasting from the hyperedge to the nodes, distributing community-level semantic information back to the item nodes and user nodes via their respective association matrices:

[0080] through After hypergraph convolution propagation, the output of the final layer is taken as the modality. Global User Representation Global Item Display The hypergraph propagation process described above is performed on both the visual and textual modalities, and the two global representations are summed element-wise to obtain the comprehensive global embedding:

[0081] in, , A global representation of the user in both visual and textual modalities. , This is the global representation of the item.

[0082] Global hypergraph encoding effectively complements bipathic local modeling: items sharing the same prototype hyperedge can exchange information through hyperedge aggregation-broadcasting, regardless of their topological distance in the interaction graph, fundamentally breaking through the receptive field limitation of local graph propagation.

[0083] Step 5: By using cross-modal alignment loss and joint optimization objective function, multi-level representation fusion of local individualized representation, inter-item latent semantic representation and community-level global semantic representation is performed and the preference prediction score is output, and the recommendation list is output.

[0084] Following global hypergraph encoding in step 4, the model obtains global user and item representations for both visual and textual modalities. These two global representations are learned independently within their respective modal hypergraphs. Direct fusion without explicit constraints can lead to semantic drift between modalities, potentially causing conflicts in the global embeddings and reducing the quality of the fused representations. This step introduces cross-modal contrastive alignment loss to semantically constrain the two global representations and integrates the multi-level representations into the final prediction score. The specific process is as follows: Step 51: Construct the cross-modal global alignment loss; Taking the user side as an example, for the current training batch users in Its visual global representation With text global representation Positive sample pairs are formed, and the global text representations of other users within the batch are used as negative samples. The alignment loss is defined in the form of InfoNCE:

[0085] in This refers to the inner product operation between vectors after L2 normalization. The temperature hyperparameter for alignment loss is used. The numerator measures the semantic consistency of the global representation of the same user across visual and textual modalities, while the denominator uses the textual global representations of all users within a batch as negative samples, forcing cross-modal representations of different users to separate from each other in the feature space. Item-side alignment loss. Positive sample pairs are formed using the visual and textual global representations of items, defined symmetrically. The comprehensive cross-modal global alignment loss is the mean of the user-side and item-side losses:

[0086] This constraint prompts the visual and text-based global hypergraph encoders to converge collaboratively towards a shared semantic alignment direction while independently modeling modal dependencies. Since the two prototype matrices were initialized in step 1 using shared large-model semantic community anchors, The optimized starting point has a better semantic benchmark, making the alignment process more efficient and robust.

[0087] Step 52: Local comparative learning; User local representation of the main graph propagation output in step 2 Partial representation of an object Apply two independent random feature masks to achieve a dropout rate. Two enhanced views are generated by randomly setting some feature dimensions to zero and then normalizing them using L2. Taking the item side as an example, from... Generate enhanced view and Comparative learning using InfoNCE loss:

[0088] in Temperature hyperparameter for local contrast loss. User-side loss. Symmetry is defined. The comprehensive local contrast loss is:

[0089] Step 53: Multi-level feature fusion; The local representation of the main graph, the latent semantic graph representation, and the L2-normalized global hypergraph representation are weighted and fused to obtain the final representation of users and items:

[0090]

[0091] in, For row-by-row L2 normalization operation, This is a global fusion weight hyperparameter used to balance the relative contributions of local individualized signals and global community semantic signals. Applying L2 normalization to the global representation eliminates amplitude differences between local and global representations, ensuring balanced utilization of both types of information in weighted fusion. The final item representation is also included. and It integrates information from two dimensions: individualized collaborative filtering signals and cross-modal item semantic associations.

[0092] Based on the final fused representation of users and items, preference prediction scores are calculated using inner product operations: Finally, the Top-K recommendation list is output, yielding the recommendation results.

[0093] Step 54: Construct a joint optimization objective; The primary optimization objective employs Bayesian Personalized Ranking (BPR) loss, learning pairwise rankings of positive and negative sample items for each user.

[0094] in To train the set of triples, and These are the observed interaction set and the sampled negative sample set, respectively. The sigmoid activation function is used. The three types of losses are linearly combined to form the final joint optimization objective:

[0095] in, and These are the balance coefficients for local contrast loss and cross-modal alignment loss, respectively. The L2 regularization coefficient is... These represent all learnable parameters of the model. The three types of losses have a clear division of labor within the joint framework: Directly optimize the primary objective of recommendation ranking; Enhance the discriminative power of local representations in the main image by comparing feature views, thereby improving robustness to sparse interaction scenarios; Cross-modal contrast constraints promote semantic consistency convergence of two-way global hypergraph representations, thereby improving the fusion quality of global embeddings.

[0096] Experimental verification In this embodiment, a dataset from a real e-commerce scenario is used for validation. Specifically, three categories from the Amazon product review dataset are used: Baby Products, Sports & Outdoors, and Clothing & Footwear. Basic statistical information is shown in Table 1. Table 1. Dataset Statistics

[0097] The experiment adopted the standard preprocessing workflow of the Amazon product dataset, with a partition ratio of 8:1:1 (training set: validation set: test set). The evaluation metrics used were Recall@20 and NDCG@20, following the full-scale ranking evaluation protocol. Table 2 presents the performance comparison results between the proposed method and existing representative methods. Table 2 Comparison of Experimental Results

[0098] Based on the experimental results in Table 2, the proposed method achieves optimal results across all evaluation metrics on the three datasets, fully validating the synergistic effectiveness of the three core designs: large-model semantic-guided initialization, inverse bias master graph denoising, and cross-modal global hypergraph alignment. Compared to the hypergraph baseline LGMRec, the proposed method improves Recall@20 by 7.93%, 10.70%, and 25.06% on the Baby, Sports, and Clothing datasets, respectively. This improvement stems from two levels: at the local modeling level, dynamic sampling of the inverse bias master graph effectively filters out the noise amplification effect of global propagation; at the global modeling level, large-model semantic-guided initialization provides a meaningful semantic starting point for the hypergraph prototype vectors, which, combined with cross-modal alignment loss constraints, fundamentally compensates for the insufficient global representation fusion quality caused by the lack of cross-modal semantic alignment in LGMRec.

[0099] Example 2 One embodiment of this disclosure provides a multimodal graph-aware item recommendation system based on large model semantic guidance, including: The semantic community guidance module is used to obtain the item category system and construct the user-item interaction graph; it uses a large model to perform global semantic analysis on the item category system, infers the semantic community description text, vectorizes the semantic community description text, and generates shared semantic initialization anchor points for the hypergraph visual and text prototype matrices. The main graph denoising module is used to measure the individual importance of the interaction edges of the user-item interaction graph, construct an inverse tendency main graph, and aggregate it using graph convolution mean pooling to obtain a local individualized representation; The Latent Semantic Graph module is used to construct a cross-modal item similarity graph and extract latent semantic representations between items; The global hypergraph encoding module is used to perform hypergraph convolution on the prototype hyperedges based on shared semantic initialization anchors to obtain a community-level global semantic representation; The feature fusion and prediction module is used to fuse local individualized representations, inter-item latent semantic representations, and community-level global semantic representations through cross-modal alignment loss and joint optimization objective function, and output preference prediction scores and recommendation lists.

[0100] Example 3 One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal graph-aware item recommendation method based on large model semantic guidance.

[0101] Example 4 One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the multimodal graph-aware project recommendation method based on large model semantic guidance.

[0102] Example 5 One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the multimodal graph perception project recommendation method based on large model semantic guidance.

[0103] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A multimodal graph-aware item recommendation method based on large-model semantic guidance, characterized in that, include: Obtain the item category system and construct a user-item interaction graph; Using a large model, a global semantic analysis of the item category system is performed to infer the semantic community description text. The semantic community description text is then vectorized and encoded to generate shared semantic initialization anchor points for both the hypergraph visual and textual prototype matrices. Individual importance measures are performed on the interaction edges of the user-item interaction graph, an inverse tendency main graph is constructed, and graph convolution mean pooling is used to aggregate the graph to obtain local individual representations; a cross-modal item similarity graph is constructed to extract latent semantic representations between items; Based on shared semantic initialization anchor points, hypergraph convolution is performed on prototype hyperedges to obtain community-level global semantic representations; By using cross-modal alignment loss and joint optimization objective function, multi-level representation fusion of local individualized representation, inter-item latent semantic representation and community-level global semantic representation is performed to output preference prediction scores and a recommendation list.

2. The multimodal graph-aware item recommendation method based on large model semantic guidance as described in claim 1, characterized in that, The process involves using a large model to perform global semantic analysis on the item category system, inferring semantic community description text, vectorizing and encoding the semantic community description text, and generating shared semantic initialization anchor points for both the hypergraph visual and textual prototype matrices. This includes: By statistically summarizing the category labels of all items in the item collection, a category distribution set at the dataset level is obtained; Using the category distribution set as input to the large model, a prompt template for semantic community induction is constructed to guide the large model to perform reasoning tasks and generate semantic community description text; The semantic community description text is encoded using a text encoder to obtain semantic community vectors. The semantic community vectors are then concatenated column by column to form a semantic community matrix. A learnable prototype matrix is ​​constructed for both the text modality and the visual modality. For the text modality prototype matrix, the semantic community matrix is ​​used directly as the initial value. For the visual modality prototype matrix, a two-stage initialization strategy is adopted. First, a linear projection is applied to the semantic community matrix to map the text semantic vectors to the visual feature space. Then, the projection result is normalized column by column to obtain the shared semantic initialization anchor point of the hypergraph visual and text prototype matrices.

3. The multimodal graph-aware item recommendation method based on large model semantic guidance as described in claim 1, characterized in that, The process of individually measuring the importance of interaction edges in the user-item interaction graph, constructing an inverse-biased main graph, and aggregating the graph using graph convolutional mean pooling to obtain a local individualized representation includes: For each edge in the user-item interaction graph, assign an individual importance weight based on the degree of the nodes it connects; Using individual importance weights as sampling probabilities, a sparse main graph is constructed by sampling the set of interaction edges with replacement using weighted probability sampling. The edge set retained by sampling is re-symmetrically normalized to construct the normalized adjacency matrix of the sparse main graph; For each user, two independent learnable embedding vectors, one visual and one textual, are maintained separately and initialized with a uniform Xavier distribution. The two embeddings are concatenated to form the initial embeddings of the user and the item. The initial embedded input sparse main graph is used to perform LightGCN-style graph convolution propagation in LP layers. The outputs of each layer are aggregated by mean pooling, and the user main graph representation and the item main graph representation are extracted from the output respectively.

4. The multimodal graph-aware item recommendation method based on large model semantic guidance as described in claim 1, characterized in that, The construction of a cross-modal item similarity map and the extraction of latent semantic representations between items include: After projecting the original pre-trained features of the items, L2 row-wise normalization is performed to obtain the normalized item feature matrix under each modality; For both text and visual modalities, cosine similarity is calculated between all item pairs to construct a fully connected semantic similarity matrix; For each item node, only the multiple neighbors with the highest similarity are retained, and a sparse semantic adjacency matrix is ​​obtained by sparsification using KNN; Weighted fusion of cross-modal semantic graphs yields a latent semantic affinity graph adjacency matrix that integrates multimodal information; Using the item splicing embedding as the initial input, graph convolution propagation is performed on the latent semantic affinity graph to output the latent semantic representation between items.

5. The multimodal graph-aware item recommendation method based on large model semantic guidance as described in claim 1, characterized in that, The process of performing hypergraph convolution on the prototype hyperedge based on shared semantic initialization anchor points yields a community-level global semantic representation, including: Based on shared semantic initialization anchors, for both text and visual modalities, the association strength between an item and a hyperedge is measured by the inner product of the item's original features and the prototype vector. The Gumbel-Softmax mechanism is used to perform differentiable multi-hyperedge soft assignment. The soft assignment vectors of all items are stacked row by row to obtain the item-hyperedge correlation matrix. The user-hyperedge association matrix is ​​derived by performing sparse message propagation on the user side of the item-hyperedge association matrix through the interaction graph. Using the main item graph as the initial input, a two-stage hypergraph convolution propagation is performed to output the global user representation and the global item representation in each modality.

6. The multimodal graph-aware item recommendation method based on large model semantic guidance as described in claim 1, characterized in that, A cross-modal global alignment loss function is constructed. The item-side alignment loss is formed by positive sample pairs of the visual and textual global representations of the item. The comprehensive cross-modal global alignment loss is the mean of the user-side and item-side values. Two independent random feature masks are applied to the user local representation and the item local representation respectively to generate two sets of enhanced views for comparative learning. The local representation of the main graph, the latent semantic graph representation, and the global hypergraph representation are weighted and fused to obtain the final representation of users and items. The preference prediction score is calculated by inner product operation.

7. A multimodal graph-aware item recommendation system based on large-model semantic guidance, characterized in that: include: The semantic community guidance module is used to obtain the item category system and build a user-item interaction graph; Using a large model, a global semantic analysis of the item category system is performed to infer the semantic community description text. The semantic community description text is then vectorized and encoded to generate shared semantic initialization anchor points for both the hypergraph visual and textual prototype matrices. The main graph denoising module is used to measure the individual importance of the interaction edges of the user-item interaction graph, construct an inverse tendency main graph, and aggregate it using graph convolution mean pooling to obtain a local individualized representation; The Latent Semantic Graph module is used to construct a cross-modal item similarity graph and extract latent semantic representations between items; The global hypergraph encoding module is used to perform hypergraph convolution on the prototype hyperedges based on shared semantic initialization anchors to obtain a community-level global semantic representation; The feature fusion and prediction module is used to fuse local individualized representations, inter-item latent semantic representations, and community-level global semantic representations through cross-modal alignment loss and joint optimization objective function, and output preference prediction scores and recommendation lists.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal graph-aware project recommendation method based on large model semantic guidance as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the multimodal graph-aware project recommendation method based on large model semantic guidance as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the multimodal graph-aware project recommendation method based on large model semantic guidance as described in any one of claims 1-6.

Images