Session recommendation method based on multi-modal hypergraph neural network

Abstract

The invention relates to the technical field of recommendation systems, in particular to a session recommendation method based on a multi-modal hypergraph neural network, which comprises the following steps of: S1, acquiring session interaction and multi-modal data (ID, image and text), and denoising multi-modal features by using behaviors so as to optimize neighborhood enhanced ID embedding and generate pure complementary representation; s2, establishing a multi-modal hypergraph, setting a precursor and a context hyperedge, and learning global high-order dependency through node-hyperedge-node attention aggregation to obtain an article representation fused with global interest; s3, adaptively weighting the ID and the multi-modal features according to the session confidence, and generating a current session representation; s4, enhancing cross-view comparison among the three views by using difficult samples, and mining fine-grained semantic association; and S5, performing attention weighting to generate session embedding, matching a next item of candidate prediction, and combining cross entropy and contrast loss training. The multi-mode and high-order relation is effectively fused; according to the method, the negative influence of sparse interaction on session recommendation performance is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of recommender system technology, and specifically to a conversational recommendation method based on a multimodal hypergraph neural network. Background Technology

[0002] Conversational recommendation aims to predict items a user might be interested in next, based solely on a series of anonymous interactions within the current session, in the absence of long-term user history information. It has been widely applied in e-commerce and content recommendation scenarios.

[0003] Most existing conversation recommendation methods are based on item ID sequences for modeling, aiming to characterize users' short-term interests by mining co-occurrence relationships or transition patterns between items. However, these methods mainly rely on discrete ID information, which makes it difficult to fully express the rich semantic information such as images and text contained in the actual display of items, thus limiting their ability to characterize users' true preferences.

[0004] To address these shortcomings, some studies have attempted to incorporate multimodal information such as images and text into conversation recommendation to enhance the expressive power of item and conversation representations. However, in practical applications, conversation recommendation scenarios generally suffer from short conversation lengths and sparse interactions. In cases of data sparsity, factors such as noise in multimodal data, mismatched modal pairs, and limited training samples can negatively impact model performance.

[0005] Furthermore, existing multimodal session recommendation methods primarily focus on modeling pairwise relationships between items, lacking effective modeling of higher-order relationships between multiple items. They also typically employ fixed multimodal fusion methods, making it difficult to adapt to the varying levels of user behavior information richness across different sessions. Simultaneously, under sparse interaction conditions, the ability to mine fine-grained semantic associations between different modalities remains limited.

[0006] Therefore, how to effectively utilize multimodal information and fully explore the higher-order relationships between items in data-sparse session recommendation scenarios to improve recommendation performance remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to address the common problems of short session length and sparse interaction in existing session recommendation scenarios. In the case of sparse data, factors such as noise information, mismatched modal pairs, and limited training samples in multimodal data can easily have a negative impact on model performance. Existing multimodal session recommendation methods mostly focus on modeling pairwise relationships between items, lack effective modeling of higher-order relationships between multiple items, and usually adopt a fixed multimodal fusion method, which is difficult to adapt to the differences in the richness of user behavior information in different sessions.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] This invention provides a session recommendation method based on a multimodal hypergraph neural network, comprising the following steps: S1, acquiring interactive behavior data of items in a session and corresponding multimodal data, including item ID information, image features, and text features; using a peer-to-peer refining module, filtering and denoising the multimodal features using item ID behavior information, and enhancing the item ID embedding using optimized multimodal neighborhood information to obtain a clean and complementary multimodal item representation; S2, constructing multimodal hypergraphs based on the multimodal item representations, establishing predecessor hyperedges and context hyperedges for item nodes in each hypergraph; learning global high-order dependencies between items through the node-hyperedge-node information transmission and attention aggregation mechanism in the hypergraph, and obtaining a fusion of global interests. Item representation; S3. Calculate the session confidence score based on the richness of historical behavior information in the session; Based on the confidence score, use a personalized modality fusion module to adaptively weight and fuse ID features and multimodal features to generate a session representation that conforms to the characteristics of the current session; S4. Introduce a cross-view contrastive learning mechanism that enhances hard samples among the item ID view, image view, and text view, and mine fine-grained semantic associations between different modalities by strengthening the learning to distinguish between hard positive samples and hard negative samples; S5. Weight items in the session based on the attention mechanism to generate the final session embedding representation, and match it with the candidate item representation to output the prediction result of the next interactive item; Train the model by jointly optimizing the cross-entropy loss function and the contrastive learning loss function.

[0010] Further, in step S1, the image features and text features corresponding to the items are extracted using a pre-trained model, and the item ID is embedded and initialized. The image features and text features are transformed into the same feature space as the item ID embedding through linear mapping. Based on the item ID behavior information in the session, element-wise multiplication is performed on the mapped image features and text features to reduce noise information in the multimodal features, resulting in a denoised multimodal feature representation. A modal affinity matrix between items is constructed based on the denoised image features and text features, and the modal neighborhood is determined by calculating the similarity between pairs of items. Based on the modal neighborhood information, neighborhood feature aggregation is performed on the item ID embedding to enhance the item ID features using multimodal neighbor information, resulting in a multimodal item representation after equivalence refinement.

[0011] Furthermore, in step S2, based on the multimodal item representation, corresponding multimodal hypergraphs are constructed in the item ID modality, image modality, and text modality, respectively, where each hypergraph uses items as nodes;

[0012] For each session, a predecessor hyperedge and a context hyperedge are constructed for the item node in the hypergraph. The predecessor hyperedge is used to connect the current item in the session with one or more of its corresponding predecessor item nodes, and the context hyperedge is used to connect multiple item nodes located within a preset sliding window range.

[0013] Further, in step S2,

[0014] The constructed hyperedges are deduplicated to form the final set of hyperedges in the multimodal hypergraph. A node-to-hyperedge feature aggregation process is performed in the hypergraph, using an attention mechanism to weighted aggregate the features of multiple nodes within the same hyperedge to obtain the corresponding hyperedge feature representation. A hyperedge-to-node feature propagation process is then performed in the hypergraph, using an attention mechanism to weighted fuse the features of multiple hyperedges associated with the same node and combine them with the node's original features to update the node's representation, thus obtaining an item representation containing global high-order dependencies.

[0015] Further, in step S3, based on the item representation, the item ID representation is concatenated with the corresponding multimodal features to obtain a joint feature representation from a modal perspective; the joint feature representation is linearly transformed and a personalized transition matrix for the corresponding modality is generated through low-rank decomposition; based on the personalized transition matrix, feature mapping and fusion operations are performed on the item representation for the corresponding modality to obtain a meta-knowledge representation from a modal perspective; based on the number of item interaction behaviors in the session and their distribution in the session, a confidence parameter reflecting the richness of the session's historical behavior information is quantitatively calculated; based on the confidence parameter, the item ID representation and the multimodal meta-knowledge representation are adaptively weighted and fused to generate a session representation for predicting the next item.

[0016] Further, in step S4, a cross-view comparison learning task is constructed among the item ID view, image view, and text view, wherein the representations of the same item in different views constitute positive sample pairs, and the representations of different items in different views constitute negative sample pairs; the similarity between positive sample pairs and negative sample pairs is calculated, and positive sample pairs with similarity below a first preset threshold are identified as difficult positive sample pairs, and negative sample pairs with similarity above a second preset threshold are identified as difficult negative sample pairs.

[0017] Furthermore, in step S4, the cross-view contrastive learning process is enhanced based on the difficult positive sample pairs and difficult negative sample pairs, and the item representations under different views are constrained by the contrastive loss function to ensure that the representations of the same item remain consistent under different views, while widening the differences in the representations of different items under different views.

[0018] Further, in step S5, a soft attention mechanism is used to assign corresponding attention weights to the representations of each item in the session, and the item representations are weighted and summed according to the attention weights to generate the final session embedding representation; the session embedding representation is matched and calculated with the representations of candidate items to obtain the prediction score of each candidate item.

[0019] Furthermore, in step S5, the candidate items are sorted according to the predicted scores, and the candidate item with the highest score is selected as the recommended result for the next interactive item; during the model training process, the cross-entropy loss function is used as the main loss function, and the contrastive loss function is combined as the auxiliary loss function to jointly optimize the model parameters.

[0020] Beneficial effects

[0021] Compared with known public technologies, the technical solution provided by this invention has the following beneficial effects:

[0022] This invention addresses the problem of sparse interaction data in multimodal session recommendation scenarios. It focuses on how to fully extract effective information under sparse data conditions and improves recommendation performance through multi-level information modeling. The invention utilizes an equivalence refinement module to filter and denoise multimodal features using historical behavior information and enhances item ID representation by combining modal neighborhood information, thereby obtaining a cleaner and more reliable data representation and providing high-quality input for subsequent modeling. By constructing a global multimodal hypergraph and designing predecessor and context hyperedges, it effectively captures global high-order dependencies between items, overcoming the limitation of relying solely on local co-occurrence relationships. By introducing a personalized modality fusion mechanism, it quantitatively calculates confidence based on the richness of historical behavior information in different sessions and adaptively adjusts the fusion weights of multimodal and ID information to meet the recommendation needs of different session modes. Finally, through a cross-view contrastive learning method enhanced with hard sample reinforcement, it mines fine-grained semantic associations between different modalities, improving the model's ability to distinguish hard samples, thus effectively mitigating the negative impact of sparse interactions on session recommendation performance. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the equivalent refining module of the present invention;

[0024] Figure 2 This is a schematic diagram of the global hypergraph module of the present invention;

[0025] Figure 3 This is a schematic diagram of the personalized modal fusion module of the present invention.

[0026] Figure 4 This is a schematic diagram of the cross-view contrastive learning method for hard sample enhancement according to the present invention.

[0027] Figure 5 This is a flowchart of the session recommendation method for the multimodal hypergraph neural network of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The following is a detailed description of a session recommendation method based on a multimodal hypergraph neural network according to the present invention, with reference to the accompanying drawings:

[0031] Example:

[0032] First, a peer-to-peer refining module was proposed (see...). Figure 1 It consists of two steps:

[0033] 1. Denoising and purifying the modal information involves using behavioral information represented by IDs to filter noise from the multimodal data. This step is called the "purifier". Figure 1 (Left side); Specifically, after obtaining image and text representations through a pre-trained model, and initializing the ID representation,

[0034] Modal information is mapped to the ID space using f(x) = σ(w2(w1x+b1)+b2). ​​ID behavioral information is then used to filter and refine the modal information, where ⊙ represents the Hadamard product. This is because image and text data often contain noise; using behavioral information to filter modal information can mitigate the negative impact of noisy data in sparse data scenarios.

[0035] 2. Enrich and enhance the ID information by using optimized multimodal representations to enhance key information in the ID features. This step is called the "enricher". Figure 1 (Right side). Next, we will build an affinity matrix for each mode. Here, 'm' represents either a visual or textual modality, specifically the cosine similarity of the modal representations of pairs of items, reflecting the proximity of their neighbors. The original ID embedding representation is enriched by finding the image and text domain neighbors for each item, thus enhancing the top-k domain information most relevant to the item's ID representation from a modal perspective. This module then yields optimized representations for the three modalities. and The optimized expression is:

[0036]

[0037] Second: The Global Hypergraph module was proposed (see...) Figure 2 This is used to fully capture the higher-order relationships between item nodes. Taking session S as an example, multimodal hypergraphs can be built for the three modal representations respectively. The left side of each gray box in the graph shows the construction method of the predecessor hyperedge, and the right side shows the construction method of the context hyperedge. Specifically, two types of hyperedges are built in each modal hypergraph: predecessor hyperedges and context hyperedges. and context hyperedge The predecessor superedge focuses on the preceding item node of the current item in the session. It identifies where each item originates from and which predecessor items might "trigger" the current item's purchase. If the current item appears multiple times in the session, all its predecessor nodes in the session are included as a set in the item's predecessor superedge.

[0038] A contextual hyperedge is used to construct a sliding window, listing the items within the window as contextual hyperedges. The aim is to focus on item transition information and observe the intent transformation process in the user's product purchase path. The sliding window length is set with three different granularities to focus on contextual content information. For each session, two types of hyperedges are created: predecessor hyperedges and contextual hyperedges. If there are overlapping hyperedges, only one is retained, resulting in a deduplicated set of hyperedges.

[0039] The feature aggregation process on the hypergraph consists of two stages:

[0040] Phase 1: From Nodes to Hyperedges. Some nodes connected by hyperedges reveal intent, i.e., the user's core preferences, but other nodes may be noise, deviating from the user's core purchase intention. Assume that nodes connected by hyperedges can form a cluster; intent is calculated as the average value of the cluster. Nodes closer to the cluster center are more likely to contain core intents. Therefore, an attention mechanism is used to guide nodes through the cluster.

[0041] They are aggregated together to obtain the corresponding hyperedge features.

[0042]

[0043] in, This indicates that node j is in the m-th mode. This represents the learnable transformation vector or weight parameters of node j in the m-th mode, and ⊙ denotes element-wise multiplication. This represents the query vector related to the target node i. The inner product is used to calculate the relevance score between node j and target node i. LeakyReLU represents a non-linear activation function that introduces non-linear expressive power. exp represents mapping the score to a positive number. ∑V i′ ∈e j This represents the summation of the relevance scores of all nodes in the hyperedge.

[0044] The second stage: From hyperedges to nodes. Based on the hyperedge features, node embeddings can be further updated. For each node, multiple hyperedges pass through it, but the degree of association between each hyperedge and the node varies. This is achieved through an attention mechanism. Greater weights are assigned to hyperedges that better match the features of the current node. During aggregation, the neighbor features are fused with the target node's own features to generate a new node representation. The global hypergraph, through information transfer and aggregation mechanisms on hyperedges, models global high-order dependencies between product nodes, alleviating the sparse interaction problem faced in local interest modeling.

[0045] The fusion expression is:

[0046] in, In the m-th mode, the hyperedge e is represented j For node v i Attention weights Represents node v i The feature representation before or intermediate updates in the m-th modality. Indicates the superedge e j In the m-th modality, q2 represents the pre-update or intermediate feature representation, q2 represents the learnable attention parameter vector, LeakyReLU represents the non-linear activation function used to enhance the model's expressive power and avoid the gradient vanishing problem, and exp represents the exponential mapping and normalization of the attention score. Represents all nodes v i The set of associated hyperedges

[0047] Third: Personalized modal fusion strategies were designed for different conversation modes. Figure 3 The right side shows the process of extracting meta-knowledge from a modal perspective, while the left side shows the quantitative calculation process of confidence scores, which reflect the richness of information about the historical behavior of a conversation.

[0048] First, meta-knowledge from a modal perspective is extracted, and two personalized transition matrices are established for visual and textual information, respectively. The two matrices are obtained by concatenating the vector embeddings of modality features, collaboration features, and conversation features, followed by linear transformation and low-rank decomposition. The knowledge transfer process involves multiplying the personalized matrix obtained in the previous step with the corresponding modality and then summing the results to obtain meta-knowledge from a comprehensive modality perspective.

[0049]

[0050] Among them, E meta Let m ∈ {v, t} represent the fused multimodal feature representation, where m ∈ {v, t} represents the summation of visual and textual features processed separately. This represents the feature representation under the m-th mode. W1 represents the learnable linear transformation parameter corresponding to the m-th mode. m Let σ(·) represent the learnable scalar weights corresponding to the m-th mode, and let σ(·) represent the nonlinear activation function.

[0051] Secondly, the confidence level (cf) is proposed. S The quantitative calculation method reflects the sufficiency of historical behavioral information in a session, guiding the tendency of subsequent fusion steps. When the historical behavioral information of a session is particularly rich, the confidence level is higher, and multimodal fusion tends to rely more on rich ID representations. If user behavior is relatively less abundant, the confidence level is lower, and multimodal information becomes more important; that is, multimodal information is needed to help predict and achieve better results. The session confidence calculation formula is:

[0052]

[0053] Where len(s) represents the number of interactive items in session s, i.e., the session length, μ represents the mean of the session length, and σ represents the standard deviation of the session length. This indicates that the session length has been standardized. `sigmoid(·)` maps the standardized values ​​to the (0,1) interval, which are then used as the confidence coefficient. s This represents the confidence level of session S; a higher value indicates more comprehensive historical behavioral information in the session.

[0054] E represents the feature representation learned based on the interaction behavior of the item ID. metaThis represents the features obtained by fusing multimodal information. This indicates that when sufficient session history is available, more emphasis is placed on ID features, (1-cf s )·E meta This indicates that when conversation history behavior is sparse, more emphasis is placed on multimodal features, E F This represents the final feature representation after fusion.

[0055] 4: Cross-view contrastive learning enhanced by difficult samples (see...) Figure 4 This involves learning fine-grained features, uncovering rich latent semantic relationships between different modal views, and capturing more granular intrinsic connections to supplement self-supervised signals. Cross-view refers to comparing each pair of images, text, and ID signals. in

[0056]

[0057] If the similarity between positive sample pairs is below each threshold, they are classified as difficult positive sample pairs, requiring further learning. Similarly, if the similarity between negative sample pairs exceeds a marginal threshold, they are considered challenging. Cross-view contrastive learning enhanced with difficult sample reinforcement not only maintains the semantic consistency of the same item across different modalities and brings semantically similar embeddings closer together, but also captures more subtle and effective self-supervised signals in sparse data scenarios, enhancing the model's sensitivity to fine distinctions and improving its ability to distinguish difficult samples.

[0058] 5. When making predictions, a soft attention mechanism is used to first generate the attention weight γ for each item in a specific session. i This is used to synthesize a conversation representation, and then the conversation is multiplied by all candidate items to select the candidate item with the highest score as the next recommended item.

[0059]

[0060] Where, γ i The output result of the i-th sample, q T Let W3 represent the learnable parameter matrix, σ represent the activation function, and W4 represent the weight matrix. Let z represent the eigenvector. i Let b represent the input features of the i-th sample, and b represent the bias term.

[0061] The main loss function is the cross-entropy loss function L. rec The auxiliary loss is the contrast loss L. cl The model performance is jointly optimized, and the final loss function is L = L rec +L cl ,

[0062]

[0063] Among them, L rec Let |I| represent the reconstruction loss, |I| represent the total number of samples, and y represent the total number of samples. i This represents the true label of the i-th sample. Represents the predicted probability of the i-th sample; when y i When = 1, the loss is caused by Dominant, (if) The closer to 1, the smaller the loss. i =0 loss by Dominant, (if more) The closer to 0, the smaller the loss.

[0064] To understand it further as:

[0065] 1. Data quality

[0066] Data quality is a crucial aspect. Introducing inappropriate multimodal information can require significant effort to correct and may even degrade the performance of the recommender system. For example, the presence of irrelevant noisy multimodal information, or naturally mismatched image-text pairs in limited training data, can hinder the model's convergence in the correct direction and limit the generation of high-quality conversation preferences. In data-sparse scenarios, the negative impact and interference from low-quality training data will be even greater. Therefore, it is necessary to process the data to obtain high-quality data for the next training step. Clean, high-quality data plays a vital role in guiding the model to predict the next item.

[0067] 2. Higher-order relations

[0068] Furthermore, previous research methods have failed to fully utilize higher-order relationships between nodes and lack exploration of robust global interests. Traditional graph neural network adjacency matrices only represent pairwise relationships between nodes, while hypergraph adjacency matrices represent relationships between nodes and hyperedges, capturing complex interactions between multiple nodes. The importance of higher-order relationships lies in their ability to capture and learn complex interactions in data. Global interests can overcome the limitations of local interests, providing additional guidance and thus better matching the user's true interests. Specifically, this study proposes a global hypergraph module to model global dependencies. Hyperedges in the hypergraph can connect more nodes, capturing more information from other nodes during neighbor aggregation. The user's global interests can overcome the limitations of local interests, providing additional guidance and thus better matching the user's true interests. Through message passing mechanisms on the hypergraph, higher-order relationships in the data can be captured and learned, thereby improving model performance.

[0069] 3. Integration Strategy

[0070] Meanwhile, since sessions typically capture only short-term behavior, user interests change dynamically across different sessions. Different sessions may exhibit different patterns, and user needs can vary significantly across sessions. Therefore, different sessions may require different recommendation strategies. For example, for sessions with ample historical user behavior information, focusing on ID interaction information during fusion can leverage historical user preferences to provide more accurate recommendations. Conversely, for sessions with sparse historical behavior, modal information should be more dominant; that is, multimodal information can help achieve better prediction results. This allows multimodal features to compensate for the lack of historical user data, providing richer session and item representations. Personalized modal fusion, which formulates recommendation strategies based on session characteristics, can adapt to these changes and better meet users' personalized needs.

[0071] 4. Semantic Relationship

[0072] Furthermore, in the context of multimodal session recommendation, data sparsity leads to models over-reliance on a small subset of salient features, neglecting fine-grained details. Contrastive learning, a special type of unsupervised learning method, aims to learn data representations by maximizing the similarity between relevant samples and minimizing the similarity between irrelevant samples. In recommendation scenarios, traditional contrastive learning only considers that the representations of the same item in different modes should exhibit high similarity, i.e., the same item in different modalities should be similar. Therefore, a cross-view contrastive learning method based on hard sample augmentation, optimized from the conventional version, is designed to focus on fine-grained intrinsic relationships. The hard sample augmentation-based contrastive learning method captures finer-grained associations between modes to supplement self-supervised signals, enhancing the model's sensitivity to fine distinctions and improving its ability to distinguish hard samples. It can utilize finer-grained semantic associations between modalities to extract self-supervised signals, enhancing the model's sensitivity to subtle features and improving its ability to distinguish hard samples, thereby mitigating the negative impact of sparse interactions.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A conversation recommendation method based on a multimodal hypergraph neural network, characterized in that, Includes the following steps: S1. Obtain interactive behavior data of items in the session and corresponding multimodal data, including item ID information, image features and text features; By using the peer-to-peer refining module, the multimodal features are filtered and denoised using item ID behavior information, and the item ID embedding is enhanced using the optimized multimodal neighborhood information, resulting in a clean and complementary multimodal item representation. S2. Based on the multimodal item representation, construct multimodal hypergraphs respectively, and establish predecessor hyperedges and context hyperedges for item nodes in each hypergraph; through the information transmission and attention aggregation mechanism of node-hyperedge-node in the hypergraph, learn the global high-order dependencies between items and obtain item representations that integrate global interests. S3. Calculate the session confidence score based on the richness of historical behavior information in the session; based on the confidence score, use a personalized modality fusion module to adaptively weight and fuse ID features and multimodal features to generate a session representation that conforms to the characteristics of the current session; S4. Introduce a cross-view contrastive learning mechanism that enhances hard samples among the item ID view, image view, and text view. By strengthening the learning to distinguish between hard positive samples and hard negative samples, fine-grained semantic associations between different modalities are explored. S5. Weight items in the session based on the attention mechanism to generate the final session embedding representation, and match it with the candidate item representation to output the prediction result of the next interactive item; train the model by jointly optimizing the cross-entropy loss function and the contrastive learning loss function.

2. The session recommendation method based on a multimodal hypergraph neural network according to claim 1, characterized in that, In step S1, The image features and text features corresponding to the items are extracted using a pre-trained model, and the item IDs are embedded and initialized. The image features and text features are transformed into the same feature space as the item ID embedding through a linear mapping. Based on the item ID behavior information in the session, element-wise multiplication is performed on the mapped image features and text features to reduce noise information in the multimodal features and obtain the denoised multimodal feature representation; Modal affinity matrices between items are constructed based on denoised image features and text features, respectively, and modal neighborhoods are determined by calculating the pairwise similarity between items; Based on the modal neighborhood information, neighborhood feature aggregation is performed on the item ID embedding to enhance the item ID features using multimodal neighbor information, thereby obtaining a multimodal item representation after peer-refined processing.

3. The session recommendation method based on a multimodal hypergraph neural network according to claim 2, characterized in that, In step S2, based on the multimodal item representation, corresponding multimodal hypergraphs are constructed in the item ID modality, image modality, and text modality, respectively, where each hypergraph uses items as nodes; For each session, a predecessor hyperedge and a context hyperedge are constructed for the item node in the hypergraph. The predecessor hyperedge is used to connect the current item in the session with one or more of its corresponding predecessor item nodes, and the context hyperedge is used to connect multiple item nodes located within a preset sliding window range.

4. The session recommendation method based on a multimodal hypergraph neural network according to claim 3, characterized in that, In step S2, The constructed hyperedges are deduplicated to form the final set of hyperedges in the multimodal hypergraph; In the hypergraph, a feature aggregation process from node to hyperedge is performed. The attention mechanism is used to perform weighted aggregation of the features of multiple nodes within the same hyperedge to obtain the corresponding hyperedge feature representation. The feature propagation process from hyperedges to nodes is performed in the hypergraph. Multiple hyperedge features associated with the same node are weighted and fused through an attention mechanism and combined with the original features of the node to update the node representation and obtain an item representation containing global high-order dependencies.

5. The session recommendation method based on a multimodal hypergraph neural network according to claim 4, characterized in that, In step S3, based on the item representation, the item's ID representation is concatenated with the corresponding multimodal features to obtain a joint feature representation from a modal perspective; The joint feature representation is linearly transformed, and a personalized transition matrix for the corresponding mode is generated by low-rank decomposition. Based on the personalized transition matrix, feature mapping and fusion operations are performed on the item representation of the corresponding modality to obtain meta-knowledge representation from the modality perspective; Based on the number of item interaction behaviors in the conversation and their distribution in the conversation, a confidence parameter reflecting the richness of the conversation's historical behavior information is quantitatively calculated. Based on the confidence parameter, the item ID representation and the multimodal meta-knowledge representation are adaptively weighted and fused to generate a session representation for predicting the next item.

6. The session recommendation method based on a multimodal hypergraph neural network according to claim 5, characterized in that, In step S4, a cross-view comparison learning task is constructed among the item ID view, image view, and text view, wherein the representations of the same item in different views constitute positive sample pairs, and the representations of different items in different views constitute negative sample pairs. Calculate the similarity between positive and negative sample pairs, and identify positive sample pairs with similarity below a first preset threshold as difficult positive sample pairs, and identify negative sample pairs with similarity above a second preset threshold as difficult negative sample pairs.

7. The session recommendation method based on a multimodal hypergraph neural network according to claim 6, characterized in that, In step S4, the cross-view contrastive learning process is enhanced based on the difficult positive sample pairs and difficult negative sample pairs, and the item representation under different views is constrained by the contrastive loss function to ensure that the representation of the same item remains consistent under different views, while widening the difference in the representation of different items under different views.

8. The session recommendation method based on a multimodal hypergraph neural network according to claim 7, characterized in that, In step S5, a soft attention mechanism is used to assign corresponding attention weights to the representations of each item in the session, and the item representations are weighted and summed according to the attention weights to generate the final session embedding representation; The session embedding representation is matched with the representation of the candidate items to obtain the prediction score for each candidate item.

9. A session recommendation method based on a multimodal hypergraph neural network according to claim 8, characterized in that, In step S5, the candidate items are sorted according to the predicted scores, and the candidate item with the highest score is selected as the recommended result for the next interactive item. During model training, the cross-entropy loss function is used as the main loss function, and the contrastive loss function is used as the auxiliary loss function to jointly optimize the model parameters.

Images