Information recommendation method, system, device, and storage medium

Abstract

The application provides an information recommendation method, system, device and storage medium, comprising: by associating a special graph structure for different expert networks, the physical and logical decoupling of modal information is realized, and the embedding feature information of the target user and the candidate item in different semantic dimensions can be more accurately described. By calculating the data routing distribution with the user-item interaction instance as the granularity, and combining the progressive routing strategy with the prior routing distribution to obtain a hybrid routing distribution, accurate expert scheduling can be realized in different context environments, that is, it can automatically identify which modal features play a key role in user decision-making in a specific scenario, thereby accurately capturing the fine-grained preferences of users. Through the gating selection mechanism, the feature contribution of key experts is retained, effectively filtering the noise interference generated by irrelevant modalities, and further improving the accuracy of the prediction score and the reliability of the recommendation result.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to an information recommendation method, system, device, and storage medium. Background Technology

[0002] Currently, multimodal data is relied upon to construct rich user and item representations to improve recommendation accuracy. However, related technologies typically employ shared attention mechanisms or simple weighted averaging to fuse features from different modalities. This fusion approach struggles to accurately capture users' fine-grained preferences in multimodal interactions, impacting the accuracy of recommendation results. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide an information recommendation method, system, device and storage medium.

[0004] Based on the above objectives, embodiments of this application provide an information recommendation method, comprising: calculating the embedding feature representations of the target user and each candidate item under a structural expert network based on an interaction graph constructed from the historical interaction behavior of a target user and multiple candidate items; calculating the embedding feature representation of each candidate item under each content expert network based on a semantic graph constructed from the similarity of content data of the multiple candidate items in multiple modalities; associating each content expert network with the semantic graph of the corresponding modality; processing the interaction context features of the target user and each candidate item using a routing network to obtain a data routing distribution; dynamically allocating the weight information of the prior routing distribution and the data routing distribution according to a time control factor to obtain a hybrid routing distribution; the time control factor decaying with the increase of optimization rounds; performing gating selection on the hybrid routing distribution to determine the gating weights of each expert network; processing the embedding feature representations corresponding to the target user and each candidate item using the gating weights to obtain a user fusion representation and an item fusion representation corresponding to each candidate item; and obtaining a prediction score for each candidate item based on the user fusion representation and the item fusion representation to obtain a recommendation result.

[0005] Optionally, based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, the embedding feature representations of the target user and each candidate item under the structural expert network are calculated; based on the semantic graph constructed from the similarity of the content data of the multiple candidate items in multiple modalities, the embedding feature representation of each candidate item under each content expert network is calculated, including: obtaining the interaction graph based on the historical interaction behavior of the target user and the candidate items; pruning the edges of the interaction graph based on the degree of the target user node and the degree of the candidate item node in the interaction graph to obtain a denoised interaction graph; and pruning the edges of the interaction graph based on the similarity of the content data of the multiple candidate items in multiple modalities. The system extracts feature information from multiple modalities of each candidate item from the data. The feature information of the multiple modalities is in the same mapping space. Based on the feature information of the multiple modalities, the feature similarity of the candidate items within each modality is calculated to obtain the semantic map of the corresponding modality. The structural expert network is used to process the denoised interaction map and its corresponding initial embedding features to obtain the embedding feature representation of the target user and the multiple candidate items under the structural expert network. The multiple content expert networks are used to process the semantic map of the corresponding modality and its corresponding initial embedding features to obtain the embedding feature representation of the multiple candidate items under the multiple content expert networks.

[0006] Optionally, the routing network includes a first multilayer perceptron and a second multilayer perceptron. The routing network is used to process the interaction context features between the target user and each candidate item to obtain a data routing distribution. This includes: processing the interaction context features between the target user and each candidate item using the first multilayer perceptron to obtain a first selection preference score for each expert network; processing the first selection preference score using the second multilayer perceptron to obtain a second selection preference score for each expert network; and normalizing the second selection preference score to obtain the data routing distribution.

[0007] Optionally, dynamically allocating weight information of the prior route distribution and the data route distribution according to a time control factor to obtain a hybrid route distribution includes: allocating weight information to the prior route distribution and the data route distribution according to the time control factor to obtain first weight information and second weight information; the first weight information is directly proportional to the time control factor, the second weight information is inversely proportional to the time control factor, and the time control factor decays as the optimization rounds increase; and performing a weighted summation of the prior route distribution and the data route distribution according to the first weight information and the second weight information to obtain a hybrid route distribution.

[0008] Optionally, gating selection is performed on the hybrid routing distribution to determine the gating weights of each expert network, including: injecting standard Gaussian noise into the hybrid routing distribution to obtain a perturbed hybrid routing distribution; selecting multiple expert networks that meet preset conditions from the perturbed hybrid routing distribution as candidate expert networks; normalizing the hybrid routing distribution of the candidate expert networks to obtain the corresponding gating weights; and setting the gating weights of the remaining expert networks to zero.

[0009] Optionally, the gating weights are used to process the embedding feature representations of the target user and each candidate item to obtain the user fusion representation and item fusion representation for each candidate item. This includes: for each candidate item, performing the following operations: obtaining the item fusion representation of the current candidate item based on the embedding feature representations of the current candidate item in the structural expert network and the multiple content expert networks, as well as the corresponding gating weights; and obtaining the user fusion representation of the target user corresponding to the current candidate item based on the embedding feature representations of the target user in the structural expert network, the embedding feature representations of the current candidate item in the multiple content expert networks, and the corresponding gating weights.

[0010] Optionally, the method further includes: updating the parameters of the expert network and the routing network with the objective of minimizing a multi-objective loss function; the multi-objective loss function is constructed based on a main task loss function, an expert load balancing loss function, and an expert stability loss function; the main task loss function is constructed based on the predicted scores of candidate items that the target user has interacted with and the predicted scores of candidate items that have not been interacted with; the expert load balancing loss function is constructed based on the gating weights of multiple candidate items on multiple expert networks; the expert stability loss function is constructed based on the embedding feature representation of the current optimization round and the embedding feature representation of the previous optimization round; and optimizing the recommendation result based on the updated parameters of the expert network and the routing network.

[0011] Another embodiment of this application provides an information recommendation system, comprising: an expert processing module, configured to calculate the embedding feature representations of the target user and each candidate item under a structural expert network based on an interaction graph constructed from the historical interaction behaviors of the target user and multiple candidate items; calculate the embedding feature representations of each candidate item under each content expert network based on a semantic graph constructed from the similarity relationships of the multiple candidate items under multiple modalities; and associate each content expert network with the semantic graph of the corresponding modality; a routing processing module, configured to process the interaction context features of the target user and each candidate item using a routing network to obtain a data routing distribution; dynamically allocate the weight information of the prior routing distribution and the data routing distribution according to a time control factor to obtain a hybrid routing distribution; and decrease the time control factor as the optimization rounds increase; and a result prediction module, configured to perform gating selection on the hybrid routing distribution to determine the gating weights of each expert network; process the embedding feature representations of the target user and each candidate item using the gating weights to obtain a user fusion representation and an item fusion representation corresponding to each candidate item; and obtain a prediction score for each candidate item based on the user fusion representation and the item fusion representation to obtain a recommendation result.

[0012] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0013] Another embodiment of this application provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the above-described method.

[0014] As can be seen from the above, the information recommendation method provided in this application introduces a hybrid expert network architecture into a multimodal graph neural network. It decouples the feature processing of different modalities by constructing multiple parallel modality-specific expert networks and employs a progressive routing mechanism to achieve instance-level dynamic modality fusion. Specifically, by associating dedicated graph structures with different expert networks, the physical and logical decoupling of modal information is achieved, significantly improving the ability to handle complex multimodal interaction scenarios and effectively solving the modality conflict problem. This allows for a more accurate characterization of the embedded feature information of target users and candidate items in different semantic dimensions. By calculating the data routing distribution at the granularity of user-item interaction instances and combining the progressive routing strategy with the prior routing distribution to obtain a hybrid routing distribution, expert scheduling can be accurately completed in different contexts. This means it can automatically identify which modal features play a key role in user decision-making in specific scenarios, thereby achieving precise targeting and capture of fine-grained user preferences. The gating selection mechanism retains the feature contributions of key experts, effectively filtering out noise interference from irrelevant modalities, further improving the accuracy of prediction scores and the reliability of recommendation results. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of an information recommendation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating an information recommendation method according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an information recommendation system according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0021] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] Information recommendation aims to personalize recommendations by analyzing users' historical behavior, explicit preferences, and implicit interactions (such as clicks, browsing, and favorites) to push items or information that users may be interested in. Recommended items can include products, videos, news, music, etc. Recommendation methods can be divided into three categories: collaborative filtering, content filtering, and hybrid recommendation. Collaborative filtering relies on user-item interaction history, making recommendations by mining the similarities between users or items; content filtering methods match item attributes (such as text descriptions, categories, and images) with user interests; hybrid recommendation methods combine multiple recommendation methods to improve the accuracy and diversity of recommendations.

[0024] However, the aforementioned recommended methods lack effective modeling of the complex coupling relationship between cross-modal semantic structure and user interaction behavior in multimodal environments. To address this issue, graph neural networks and cross-modal attention mechanisms can be introduced to unify the modeling of relationships between users, items, and their multimodal attributes. Graph neural networks are deep learning models specifically designed for processing graph-structured data. Their core idea is to learn node representations and the overall structural features of the graph by using message passing mechanisms to pass and aggregate information between nodes and edges. The process may include node embedding initialization, message passing, neighbor information aggregation (such as summation, averaging, or taking the maximum value), and node feature updates. Cross-modal attention is a technique in multimodal data processing that enhances information interaction and fusion between different modalities through attention mechanisms. Its core idea is to dynamically focus on and utilize relevant information from other modalities when processing data of a particular modality, thereby improving overall performance. The process may include feature extraction, attention calculation, information fusion, and decision output. Through cross-modal attention, the model can more effectively integrate multimodal information.

[0025] In recommendation processes based on graph neural networks and cross-modal attention mechanisms, related techniques typically employ shared attention mechanisms or static weighted averaging strategies to fuse embedding vectors from different modalities such as text, images, and audio. However, although graph structures significantly enhance node representation capabilities by modeling user-item interactions, several key limitations remain in the multimodal feature fusion stage: Firstly, there is the problem of modal contribution ambiguity, i.e., it is difficult to accurately determine which modalities are truly effective for the current recommendation decision, leading to redundant or even noisy modal interference in the final representation and reducing recommendation accuracy. Secondly, there is the problem of modal entanglement, i.e., semantic signals from different modalities interfere with each other in the latent space, weakening the ability to express fine-grained user preferences.

[0026] Therefore, even though graph neural networks improve the depth and breadth of relationship modeling, they still struggle to accurately capture users' true intentions in multimodal interactions, affecting the accuracy of recommendations.

[0027] Based on this, this application provides an information recommendation method that introduces a hybrid expert network architecture into a multimodal graph neural network. By constructing multiple parallel modality-specific expert networks, the feature processing of different modalities is decoupled, and a progressive routing mechanism is employed to achieve instance-level dynamic modality fusion. Specifically, by associating dedicated graph structures with different expert networks, the physical and logical decoupling of modal information is achieved, significantly improving the ability to handle complex multimodal interaction scenarios and effectively resolving modality conflict issues. This allows for a more accurate characterization of the embedded feature information of target users and candidate items in different semantic dimensions. By calculating the data routing distribution at the granularity of user-item interaction instances and combining the progressive routing strategy with the prior routing distribution to obtain a hybrid routing distribution, expert scheduling can be accurately completed in different contexts. This means that it can automatically identify which modal features play a key role in user decision-making in specific scenarios, thereby achieving precise targeting and capture of fine-grained user preferences. A gating selection mechanism preserves the feature contributions of key experts, effectively filtering out noise interference from irrelevant modalities, further improving the accuracy of prediction scores and the reliability of recommendation results.

[0028] refer to Figure 1 This diagram illustrates an application scenario of the information recommendation method provided in this application. The application scenario may include a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks. The terminal device 101 includes, but is not limited to, desktop computers, mobile phones, mobile computers, tablets, media players, smart wearable devices, personal digital assistants, or other electronic devices capable of information display and interaction. The server 102 and data storage system 103 can both be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, big data, and artificial intelligence platforms.

[0029] Server 102 can be used to provide personalized information recommendation services to users of terminal device 101 based on a trained recommendation model. This recommendation model may include multiple expert networks, each responsible for processing a specific type of information. For example, a visual expert network may focus on analyzing aesthetic preferences in image feature information, a text expert network may focus on analyzing functional requirements in text feature information, and a structural expert network may focus on mining collaborative patterns in group behavior. Based on this, the recommendation process may include: calling the trained recommendation model; constructing an interaction graph based on the historical interaction behavior of the target user and multiple candidate items; calculating the embedding feature representations of the target user and each candidate item under the structural expert network; constructing a semantic graph based on the similarity of content data of multiple candidate items under multiple modalities; calculating the embedding feature representation of each candidate item under each content expert network; and associating each content expert network with the semantic graph of its corresponding modality. A routing network is used to process the interaction context features to obtain a data routing distribution, which is then combined with the prior routing distribution to generate a hybrid routing distribution. Furthermore, a gating selection process is used to determine the candidate expert networks and their gating weights, thereby fusing the user fusion representation and the item fusion representation and calculating the prediction score. Server 102 can send the recommendation results generated based on the predicted score to terminal device 101, and terminal device 101 can display the recommended content to the user to achieve accurate information distribution.

[0030] The recommendation timing can be during a user's browsing of information on a terminal device, during interaction with the terminal device (such as clicking, searching, or lingering), or dynamically triggered in multiple scenarios such as when the user launches an application, completes a key operation, or exhibits potential interest signals, thereby achieving personalized recommendations. The candidate item set corresponding to different recommendation timings may be different. For example, if the target user is lingering on an item details page, the candidate items may be complementary products, competing products, or frequently viewed / purchased items related to the current item on that page; if the target user is browsing the homepage information stream, the candidate items may come from popular or personalized content matched with their long-term interest profile; if the target user has just completed a search, the candidate items mainly consist of products or content semantically related to the search keywords. In an optional embodiment, before calling the recommendation model to execute the recommendation service, user interaction behavior data can be obtained to dynamically determine the recommendation trigger timing and generate a candidate item set accordingly, thereby obtaining the corresponding interaction graph, semantic graph, and interaction context features. The user identification information, interaction behavior data, interaction graph, semantic graph, and interaction context features can be stored in association in the data storage system 103.

[0031] The data storage system 103 can store training data, each training data point including user historical behavior data, item content data across multiple modalities, constructed interaction graphs and semantic graphs, and interaction context features. The server 102 can jointly train the expert network, routing network, and gating mechanism in the recommendation model based on this training data. This enables the recommendation model to automatically select appropriate expert networks and assign appropriate weights based on different interaction context features, achieving precise targeting and recommendation of fine-grained user preferences. The sources of training data include, but are not limited to, existing databases, data crawled from the internet, or real-time interaction data generated when users use the client. When the prediction accuracy of the recommendation model reaches the required level, the server 102 can provide online recommendation services based on the recommendation model, and update the interaction graph, semantic graph, and interaction context features in the data storage system 103 based on newly added interaction data, newly added candidate items, and / or newly added target users, continuously optimizing the parameters of each network in the recommendation model.

[0032] When server 102 receives the same interaction behavior data generated by the same user, it can obtain the latest interaction graph, semantic graph and interaction context features from data storage system 103 to provide recommendation services for that user.

[0033] The information recommendation model of this application embodiment can be applied to various scenarios such as e-commerce recommendation, short video distribution, social content push, and news information distribution. It can train the network in the recommendation model based on training data from different industry fields to obtain recommendation models suitable for different vertical fields. The following is combined with Figure 1 The application scenarios described above illustrate the training and recommendation methods for the information recommendation model according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0034] See Figure 2 As shown, an information recommendation method is provided, which may include: S1. Based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, calculate the embedding feature representation of the target user and each candidate item under the structural expert network; based on the semantic graph constructed from the similarity of the content data of multiple candidate items in multiple modalities, calculate the embedding feature representation of each candidate item under each content expert network; each content expert network is associated with the semantic graph of the corresponding modality.

[0035] S2. Process the interaction context features between the target user and each candidate item using the routing network to obtain the data routing distribution; dynamically allocate the weight information of the prior routing distribution and the data routing distribution according to the time control factor to obtain the hybrid routing distribution; the time control factor decays as the optimization rounds increase.

[0036] S3. Perform gating selection on the hybrid routing distribution to determine the gating weights of each expert network; using the gating weights, process the embedded feature representations corresponding to the target user and each candidate item to obtain the user fusion representation and item fusion representation corresponding to each candidate item.

[0037] S4. Based on the user fusion representation and the item fusion representation, obtain the prediction score between each candidate item to obtain the recommendation result.

[0038] In some embodiments, the target user may refer to the user to whom personalized recommendations are to be made. Candidate items may refer to items that the target user might be interested in; these can be all items or items obtained through initial screening of all items, such as based on rule-based filtering, popularity, geographic restrictions, category preferences, etc. The candidate items for different target users may be the same or different. The final recommendation result is obtained through further screening from multiple candidate items.

[0039] Before constructing the interaction graph and semantic graph, content data of candidate items and interaction behavior data of target users can be received and preprocessed to obtain embedded feature representations. For example, when the content data is image data of candidate items (such as main product images and detail images), the image data can be standardized, such as by unifying resolution and pixel normalization, and converted into high-dimensional machine vision feature information. This feature information contains information such as the aesthetic style and visual texture of the item. When the content data is text data of candidate items (such as titles, categories, and detailed descriptions), the text data can be cleaned (such as removing irrelevant characters) and segmented, and converted into high-dimensional natural language feature information. This feature information contains information such as the functional parameters and semantic categories of the product. Based on the received historical interaction behaviors (such as clicks and purchase records), an interaction graph reflecting user preferences and group collaboration relationships can be constructed.

[0040] Embedded feature representation refers to the semantic vector representation of the target user or candidate item in a given modality, learned by each expert network based on its corresponding modality association graph. In this embodiment, different modalities may correspond to different association graphs. Each expert network is specifically responsible for processing data of a particular modality, such as text, images, videos, audio, and interactions, and is associated with the association graph of that modality. A modality association graph can refer to a graph structure constructed for a specific modality, where nodes represent entities in that modality and edges represent semantic relationships between nodes in that modality. Specifically, the modality association graph of the structural expert network is a user-item interaction graph constructed based on historical interaction behavior data. By modeling the collaborative relationship between the target user and candidate items based on the user-item interaction graph, the modality association graph of each content expert network is a semantic graph constructed based on the feature similarity of multiple candidate items in the corresponding modality, used to capture fine-grained semantic information in the corresponding modality. Different content expert networks are associated with semantic graphs of different modalities. For example, when the content expert network is a visual expert network, the association graph is a visual semantic graph; when the content expert network is a text expert network, the association graph is a text semantic graph. Therefore, the target user can correspond to the semantic vector representation of the structural expert network, and each candidate item can correspond to multiple semantic vector representations from different expert networks. These representations reflect the preferences of the candidate item in different modalities.

[0041] Interaction context features refer to auxiliary features introduced in addition to interaction behavior when predicting a target user's interest in candidate items, used to comprehensively characterize the decision-making environment of the target user's interaction with candidate items. In this embodiment, interaction context features can be obtained based on the initial embedding features of the target user and the initial embedding features of the candidate items. In addition, they can be combined with information such as the context features of the interaction scenario, the category attribute features of the candidate items, and the availability features of the candidate items in each modality to enhance the adaptability of the routing network to different business scenarios. Initial embedding features refer to the original vector representation before interaction, which can be understood as the embedding features formed by embedding and encoding the basic information of the target user or candidate items. In this embodiment, initial embedding features can be obtained by embedding based on the identity document (ID) information of the target user or candidate items, or by concatenating or weighting the ID embedding with the multimodal data embedding.

[0042] A routing network can refer to a learnable, lightweight quantum network used to dynamically generate data routing weights for each expert network based on the interaction context features between the target user and candidate items. These data routing weights reflect the contribution of each modal expert to the recommendation result in the current interaction context, and different expert networks can be assigned different weights. Each candidate item corresponds to an independent data routing distribution when interacting with a specific target user; by performing routing decisions at the granularity of user-item interaction instances, the routing network can capture the fine-grained interests and preferences of the target user in multimodal information.

[0043] The prior route distribution can be constructed based on the binding relationship between expert networks and modal association graphs, as well as the available modal information of specific user-item interaction instances. In one implementation, an initial weight distribution can be pre-assigned to multiple expert networks based on the modal availability of a candidate item. Modal availability refers to whether there is usable information for a candidate item in different modalities. For example, candidate item A has a product title (text) and main image (image), but no interaction data (structure) with the target user; that is, the text modality and image modality of candidate item A are available, but the structure modality is not. Modal availability can be represented in the form of a binary mask or confidence score to obtain the prior route distribution. For example, when the image information of the candidate item is available and more reliable, the prior route distribution can be set. (Visual / Textual / Structured); When textual information dominates, prior route distribution can be set. When content modalities are missing or unreliable, prior route distribution can be set. Here, 1 indicates that the mode is available, and 0 indicates that the mode is not available.

[0044] After obtaining the prior route distribution and the data route distribution, the two can be combined to obtain a hybrid route distribution. This hybrid route distribution takes into account modal feasibility and real-time user preferences, avoids assigning weights to unavailable modalities, and retains personalized expressive capabilities.

[0045] To address the issues of "expert collapse" (where all candidate items are routed to the same expert) or "random routing" that easily occur in the early stages of training of hybrid expert networks due to random parameter initialization, a progressive learning mechanism can be introduced. This mechanism introduces a control factor that decays over time. In the early stages of training, it forces the detection model to follow the prior routing distribution, ensuring that each expert receives sufficient gradient updates. As training progresses, control is gradually released, smoothly transitioning the weights of the routing network from "prior rule guidance" to "data-driven adaptation." Combined with gating selection, this effectively solves the problems of difficult training and easy collapse in hybrid expert networks.

[0046] Gating selection can refer to selecting multiple candidate expert networks from multiple expert networks based on a hybrid routing distribution, and normalizing their corresponding routing weights to generate the final gating weights. The gating weights of unselected expert networks in user-item interaction instances can be set to 0 to improve efficiency and focus. Gating selection can employ methods such as Top-K filtering and threshold truncation, which are not limited here. In the embodiments of this application, different user-item interaction instances can correspond to different candidate expert networks and different gating weights. That is, for each specific user-item interaction instance, a suitable combination of expert networks can be dynamically selected for feature fusion. This not only achieves fine-grained personalized recommendations but also significantly enhances the robustness of the recommendation model in handling cold-start items and modality conflict scenarios.

[0047] By utilizing gating weights, the embedded feature representations corresponding to the target user and each candidate item are processed to obtain a user fusion representation and an item fusion representation for each candidate item. The user fusion representation can be used to characterize preferences, while the item fusion representation can be used to characterize basic attributes or content and collaborative features. Combining the user fusion representation and the item fusion representation of each candidate item yields a predicted score for each candidate item. Based on the predicted scores of all candidate items, the items ultimately recommended to the target user are selected from all candidate items.

[0048] In this embodiment, a hybrid expert network architecture is introduced into a multimodal graph neural network. Multiple modality-specific expert networks are constructed in parallel to decouple the feature processing of different modalities, and a progressive routing mechanism is employed to achieve instance-level dynamic modality fusion. Specifically, by associating dedicated graph structures with different expert networks, the physical and logical decoupling of modal information is achieved, significantly improving the ability to handle complex multimodal interaction scenarios and effectively resolving modality conflict issues. This allows for a more accurate characterization of the embedded feature information of target users and candidate items in different semantic dimensions. By calculating the data routing distribution at the granularity of user-item interaction instances and combining the progressive routing strategy with the prior routing distribution to obtain a hybrid routing distribution, expert scheduling can be accurately completed in different contexts. This means that it can automatically identify which modal features play a key role in user decision-making in specific scenarios, thereby achieving precise targeting and capture of fine-grained user preferences. A gating selection mechanism preserves the feature contributions of key experts, effectively filtering out noise interference from irrelevant modalities, further improving the accuracy of prediction scores and the reliability of recommendation results.

[0049] In one optional embodiment, based on an interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, the embedding feature representations of the target user and each candidate item under the structural expert network are calculated; based on a semantic graph constructed from the similarity of the content data of multiple candidate items in multiple modalities, the embedding feature representation of each candidate item under each content expert network is calculated; each content expert network is associated with the semantic graph of the corresponding modality, which may include: S100. Based on the historical interaction behavior between the target user and the candidate items, obtain the interaction graph; based on the degree of the target user node and the degree of the candidate item node in the interaction graph, prune the edges of the interaction graph to obtain a denoised interaction graph.

[0050] An initial bipartite interaction graph can be constructed based on the historical behavior data of the target user and candidate items. Considering that there may be accidental touches or non-preference interaction noise in the real historical behavior data, the initial bipartite interaction matrix can be processed according to a degree-sensitive edge pruning strategy, specifically: For each edge in the initial bipartite interaction graph The retention probability can be calculated based on the degree of the target user node and the degree of the candidate item node. Its expression can be:

[0051] in, Can represent target user node The degree, Can represent candidate item nodes The degree, For smoothing terms, the higher the degree of the expression, the more likely the corresponding node is a popular item or an active user, and the higher the probability that its edges are considered noise or herd behavior, thus increasing the probability of it being pruned.

[0052] Based on the above operations, a dynamically denoised interaction graph can be constructed, and different interaction graphs can be corresponding to different target users.

[0053] S101. Based on the content data of multiple candidate items in multiple modalities, extract the feature information of each candidate item in multiple modalities; the feature information of multiple modalities is in the same mapping space.

[0054] Feature extraction can be performed on the multimodal data of candidate items to obtain the original feature information of the corresponding modality. Multimodal data can include, but is not limited to, image content, text content, video content, etc. Image content can include image files, image links, etc., and text content can include product titles, descriptions, attributes, etc. For example, for items... The original visual features can be extracted from the image content using pre-trained convolutional neural networks (such as ResNet-50). The original text features are extracted from the text content using a pre-trained language model (such as Sentence-BERT). .

[0055] Considering that the feature dimensions of different modalities may be inconsistent, different learnable linear projection matrices can be used to process the original features of multiple modalities separately to achieve feature space alignment. This ensures that the processing results are in the same mapping space, meaning that the feature information of multiple modalities after processing are vector representations of the same length in the same coordinate system, facilitating subsequent similarity calculation and graph propagation. For example, two independent linear projection matrices can be introduced. and Regarding the aforementioned original visual features and original text features The process is performed to obtain the aligned feature information. and Its expression is:

[0056] in, In Feature dimensions that can represent original visual features In Feature dimensions that can represent the features of the original text In The feature dimension can represent the learnable projection matrix corresponding to the original visual features. In It can represent the feature dimension of the learnable projection matrix corresponding to the original text features. The feature dimension of the aligned feature information.

[0057] S102. Based on the feature information of multiple modalities, calculate the feature similarity of candidate items within each modality to obtain the semantic map of the corresponding modality.

[0058] After obtaining the aligned feature information, in order to mine potential semantic relationships between items and provide independent propagation paths for subsequent expert networks of corresponding modalities, homogeneous graphs for each modality can be constructed. In this homogeneous graph, nodes represent candidate items, and edges represent the similarity relationships between different candidate items. These similarity relationships can be obtained by calculating cosine similarity. For example, candidate items... With candidate items In a specific mode The expression for cosine similarity can be:

[0059] in, Can represent candidate items and candidate items In modality The similarity score below Can represent candidate items In modality The following feature information, Can represent candidate items In modality The following feature information.

[0060] In this embodiment, the homogeneous graph can be directly used as the semantic graph of the modality, or it can be further processed before being used as the semantic graph of the modality. In one implementation, to prevent the homogeneous graph from being too dense, leading to excessive computational complexity and introducing irrelevant noise, a Top-K sparsity strategy can be performed on the homogeneous graph to retain the graph with the highest similarity. 1 neighbor, obtain semantic graph Its expression can be:

[0061] The above expression can be understood as: if candidate items In candidate items In the Top-K similarity list, if there is an edge connecting the two candidate item nodes, it is denoted as 1; otherwise, there is no edge between the two candidate item nodes, denoted as 0. Semantic graph The first in Rows can represent candidate items What are some similar candidate items for the connection?

[0062] Furthermore, to reduce computational overhead and achieve a low-pass filtering effect, this semantic graph... Once constructed, the object-to-object semantic graph can be frozen. During subsequent model training, its adjacency relationships remain unchanged and it does not participate in gradient updates. The frozen object-to-object semantic graph and the denoised user-to-object interaction graph constructed above provide topological support for multiple subsequent expert networks.

[0063] S103. Use a structural expert network to process the denoised interaction graph and its corresponding initial embedding features to obtain the embedding feature representations of the target user and multiple candidate items under the structural expert network; use multiple content expert networks to process the semantic graphs of their corresponding modalities and their corresponding initial embedding features to obtain the embedding feature representations of multiple candidate items under multiple content expert networks.

[0064] The initial embedding features corresponding to the interaction graph can be obtained based on the basic information of the target user or candidate items; the initial embedding features corresponding to the semantic graph can be the feature information of the corresponding modality.

[0065] After constructing the multimodal association graph (which can be understood as the semantic graph and interaction graph mentioned above), a dedicated expert network capable of processing different association graphs can be established. Each expert network performs information propagation and parameter updates on its corresponding association graph. Different expert networks do not share graph structures, structurally ensuring that the processing of each modality is isolated from each other. For example, a hybrid expert network architecture containing three types of experts is constructed: a visual expert network, a text expert network, and a structural expert network; and each type of expert network is physically bound to a specific association graph structure. The binding relationship can be as follows: Visual Experts Network : Can be bound to frozen visual semantic graphs This is specifically responsible for aggregating features of visually similar items. The initial embedding features of each candidate item node in this visual semantic graph can be the corresponding visual feature information. Different candidate item nodes can correspond to different visual feature information.

[0066] Text Expert Network : Can be bound to frozen text semantic graphs This is specifically responsible for aggregating features of similar items in the text semantic graph. The initial embedding features of each candidate item node in the text semantic graph can be the corresponding text feature information. Different candidate item nodes can correspond to different text feature information.

[0067] Structural Experts Network : Can be bound to denoised interactive graphs It is specifically responsible for mining collaborative filtering signals between users and items. The initial embedding features of the target node in this interaction graph can be obtained based on the basic information of the target user, and the initial embedding features of the candidate item node can be obtained based on the basic information of the corresponding candidate item. For details, please refer to the above content, which will not be elaborated here.

[0068] Based on visual feature information, a visual expert network is used to process the associated visual semantic graph to obtain the embedding feature representation of each candidate item under the visual expert network. Based on textual feature information, a text expert network is used to process the associated text semantic graph to obtain the embedding feature representation of each candidate item under the text expert network. Based on interaction feature information, a structural expert network is used to process the associated interaction graph to obtain the embedding feature representations of the target user and each candidate item under the structural expert network.

[0069] In this embodiment, on the one hand, by constructing and freezing a similarity-based item-item semantic graph, the low-pass filtering characteristics of the graph convolutional network are utilized to effectively filter out high-frequency noise in the original modal features and provide rich semantic neighbor information for cold-start items. On the other hand, by constructing a denoised user-item interaction graph and introducing a degree-sensitive edge pruning strategy, false trigger noise and herding behavior in the interaction data are dynamically removed, significantly enhancing the structural expert's ability to capture real collaborative signals. The synergistic effect of the dual-graph structure provides a solid and high-quality data foundation for the learning of the upper-layer multimodal expert network, further improving the ability to handle complex multimodal interaction scenarios and increasing the efficiency of calculating recommendation results.

[0070] In one optional embodiment, the structural expert network is used to process the denoised interaction graph and its corresponding initial embedding features to obtain the embedding feature representations of the target user and multiple candidate items under the structural expert network. This may include: using the structural expert network, performing multi-layer information propagation on the denoised interaction graph according to the initial embedding features corresponding to the denoised interaction graph to obtain the embedding feature representation of each node in the denoised interaction graph; and performing mean pooling on the embedding feature representations of each node in the denoised interaction graph to obtain the embedding feature representations of the target user and multiple candidate items under the structural expert network.

[0071] Multiple content expert networks are used to process the semantic graphs of their corresponding modalities and their corresponding initial embedding features to obtain the embedding feature representations of multiple candidate items under multiple content expert networks. This includes: for each content expert network, multi-layer information propagation is performed on the semantic graph based on the initial embedding features corresponding to the semantic graph of its corresponding modality to obtain the embedding feature representation of each node in the semantic graph; mean pooling is performed on the embedding feature representation of each node in the semantic graph to obtain the embedding feature representation of multiple candidate items under the structural expert network.

[0072] In this embodiment, each expert network can use a lightweight graph convolution mechanism to propagate information on its respective bound graph structure. The input to each expert network is the initial embedded features on its bound graph. This lightweight graph convolution mechanism does not introduce additional nonlinear transformations and complex parameters, but instead performs multi-layer neighbor feature weighted averaging and accumulation along the adjacency relationship to obtain a multi-layer propagation representation for each node.

[0073] For the A network of experts, its first The feature representation of a layer can be calculated as follows:

[0074] in, The information propagation layer index of graph convolution can be represented when hour, For the first The initial embedding matrix of an expert network; when hour, It can be passed through The node representation after aggregation of next-neighbors; To propagate the total number of layers hyperparameter. It can be represented as the first A semantic graph of expert network bindings.

[0075] For example, the semantic graph bound to the visual expert network and the text expert network contains only a set of item nodes. Therefore, the propagation of these two types of expert networks only occurs between item nodes. The denoised interaction graph bound to the structural expert network contains a set of user nodes. With item node set Therefore, the adjacency relationship of the target user node in the structural expert network comes from the set of items that it has interacted with, and information propagation can be completed through the normalized adjacency matrix, thus obtaining the embedded feature representation of the target user in the structural expert network; the adjacency relationship of the candidate item node in the structural expert network comes from the connected target node and candidate item node, thus obtaining the embedded feature representation of each candidate item in the structural expert network.

[0076] go through After layer propagation, each expert network outputs its final embedded feature representation. To avoid over-smoothing, a hierarchical readout mechanism can be used to perform mean pooling on the embeddings of each layer:

[0077] Through this step, the content expert network can output an embedded feature representation with semantic consistency, while the structural expert network can output an embedded feature representation with collaborative relevance.

[0078] In one optional embodiment, the routing network includes a first multilayer perceptron and a second multilayer perceptron. The routing network is used to process the interaction context features between the target user and each candidate item to obtain a data routing distribution. This process may include: processing the interaction context features between the target user and each candidate item using the first multilayer perceptron to obtain a first selection preference score for each expert network; processing the first selection preference score using the second multilayer perceptron to obtain a second selection preference score for each expert network; and normalizing the second selection preference score to obtain the data routing distribution.

[0079] To enable dynamic scheduling of expert networks with different modalities, a lightweight instance-level routing network can be constructed. This routing network can be based on user-item interaction instances. For granular execution, specifically, the routing network can receive the interaction context characteristics of the instance. It outputs the unnormalized choice preference score vectors for each expert network. .in, For the number of expert networks.

[0080] The routing network can be designed as a two-layer, multi-layer perceptron, and its computational expression can be:

[0081] in, For interactive instances Interaction context features; The weight matrix of a learnable first-layer perceptron can be represented. The weight matrix of a learnable second-layer perceptron can be represented. This can represent the bias term of the first multilayer perceptron. This can represent the bias term of the second-layer perceptron. It can represent a nonlinear activation function; This is the score for the second choice preference in the output.

[0082] Because the output layer dimension is set to the number of experts ,therefore The Wei and Di Each expert is paired one-to-one, and its numerical value characterizes the strength of the unnormalized preference selected by the expert network in the current interaction instance; for Softmax can be used to obtain the data routing distribution of each expert network. The output of this routing network (or This will serve as the foundational input for subsequent progressive course routing strategies.

[0083] In one optional embodiment, dynamically allocating weight information of the prior route distribution and the data route distribution according to a time control factor to obtain a hybrid route distribution may include: allocating weight information to the prior route distribution and the data route distribution according to the time control factor to obtain first weight information and second weight information; the first weight information is directly proportional to the time control factor, the second weight information is inversely proportional to the time control factor, and the time control factor decays as the optimization rounds increase; and performing a weighted summation of the prior route distribution and the data route distribution according to the first weight information and the second weight information to obtain the hybrid route distribution.

[0084] In this embodiment of the application, in order to quantify the training process and control the ratio of "guided" to "adaptive" training, a time control factor that decays exponentially with the number of optimization rounds can be defined. In the first In each optimization round The calculation expression can be:

[0085] in, This can represent a preset attenuation rate hyperparameter, for example, , Optimize the index for the current round.

[0086] Based on the time control factor, weight information is assigned to the prior route distribution and the data route distribution to obtain the hybrid route distribution. In optimizing the rounds When the mixed route distribution is calculated, the expression can be:

[0087] In this expression, the time control factor It can represent the first weight information. It can represent the second weight information.

[0088] In the initial stage of optimization (e.g., )hour, At this stage, the primary control is based on the prior route distribution to prevent expert network collapse and ensure that each expert network performs its respective function. As the number of optimization iterations increases, i.e. Increase This means gradually relinquishing control and shifting the routing towards data routing distribution to learn better adaptive strategies.

[0089] In one optional embodiment, gating selection of the hybrid routing distribution to determine the gating weight of each expert network may include: injecting standard Gaussian noise into the hybrid routing distribution to obtain a perturbed hybrid routing distribution; selecting multiple expert networks that meet preset conditions from the perturbed hybrid routing distribution as candidate expert networks; normalizing the hybrid routing distribution of the candidate expert networks to obtain the corresponding gating weights; and setting the gating weights of the remaining expert networks to zero.

[0090] To reduce computational redundancy and filter out noisy expert networks, hybrid routing distribution will be used. As a gating input, a sparse Top-K gating selection is performed on it.

[0091] To facilitate expert load balancing and increase exploratory capabilities, a hybrid routing distribution can be implemented during the training phase. Inject standard Gaussian noise The perturbed hybrid route distribution is obtained. :

[0092] Further alternatively, it can be Perform non-negative truncation or renormalization to ensure the stability of subsequent gating.

[0093] Furthermore, it can be retained The top-K experts with the highest scores are selected, and the weights of the remaining experts are forced to 0. The top-K expert network is the candidate expert network, and its index set can be denoted as... The sparsified hybrid route distribution is then:

[0094] In the set The non-zero weights are then subjected to Softmax normalization to obtain the gating weights of the candidate expert network. :

[0095] In an optional embodiment, the gating weights are used to process the embedded feature representations of the target user and each candidate item to obtain the user fusion representation and item fusion representation for each candidate item. This may include: For each candidate item, the operations include: obtaining the item fusion representation of the current candidate item based on the embedding feature representation of the current candidate item in the structural expert network and the multiple content expert networks, as well as the corresponding gating weights; and obtaining the user fusion representation of the target user corresponding to the current candidate item based on the embedding feature representation of the target user in the structural expert network, the embedding feature representation of the current candidate item in the multiple content expert networks, and the corresponding gating weights.

[0096] Using the calculated sparse gating weights Embedded feature representations of the outputs of each expert network Perform a weighted summation to obtain the multimodal fusion representation of the current node (target user or candidate item). Its expression can be:

[0097] Should It integrates complementary information from multiple modes and eliminates modal noise through sparse routing.

[0098] In one optional embodiment, obtaining a prediction score based on the user fusion representation and the item fusion representation to obtain a recommendation result may include: calculating the inner product of the user fusion representation and the item fusion representation to obtain a recommendation score between the target user and each candidate item; sorting all candidate items from high to low according to the recommendation score, and selecting multiple candidate items that meet the requirements to generate a recommendation list.

[0099] For target users and candidate items It can be fused according to the user's representation. Integration with items Perform the inner product operation to obtain the prediction score. Its expression can be:

[0100] For the same target user For each candidate item Calculate each once After obtaining a set of scores, the Top-K items are selected by sorting them from highest to lowest. This predicted score quantifies the comprehensive matching degree between the target user and the candidate items in the multimodal semantics and collaborative filtering space. If the inner product of the user fusion representation and the fusion representation of a certain item is larger, it indicates that the candidate item is more in line with the target user's interests in terms of content semantics and collaborative signals, and therefore has a higher score and is ranked higher.

[0101] In an optional embodiment, the method may further include: updating the parameters of the expert network and the routing network with the objective of minimizing the multi-objective loss function; the multi-objective loss function is constructed based on the main task loss function, the expert load balancing loss function, and the expert stability loss function; the main task loss function is constructed based on the predicted scores of candidate items that the target user has interacted with and the predicted scores of candidate items that have not been interacted with; the expert load balancing loss function is constructed based on the gating weights of multiple candidate items on multiple expert networks; the expert stability loss function is constructed based on the embedding feature representation of the current optimization round and the embedding feature representation of the previous optimization round; and optimizing the recommendation result based on the updated parameters of the expert network and the routing network.

[0102] To ensure recommendation accuracy and address the common issues of uneven load and training instability in hybrid expert networks, a multi-objective joint loss function consisting of the following three parts can be constructed.

[0103] Main Task Loss Function: To optimize recommendation ranking, a pairwise learning strategy can be employed. For each learning pair containing positive and negative sample items, the difference between the predicted scores of the positive and negative samples is calculated. Activation processing and log-likelihood calculation are then performed on this difference to construct the main task loss. Minimizing this main task loss is the objective of optimizing recommendation ranking. Wherein, positive sample items... Can refer to candidate items that the target user has interacted with, and negative sample items. This can refer to candidate items that the target user has not interacted with. The main task loss function... The expression can be:

[0104] in, This can be used as a training set. It can be the Sigmoid function. Possible positive sample prediction scores, The negative sample prediction score can be used. This main task loss function encourages positive samples to have higher prediction scores than negative samples.

[0105] Expert load balancing loss function: This loss function is designed to prevent some expert networks from being overused while others remain idle. Based on the average gating weights of all expert networks across all candidate items, the standard deviation and mean are calculated to obtain the coefficient of variation. The square of this coefficient of variation is used to construct the expert load balancing loss function. By minimizing this loss function, the utilization rate of all expert networks tends to be uniform, thus ensuring expert load balancing. The expression can be:

[0106] in, The average gating weight of all expert networks across all candidate items; This can represent the standard deviation of the average gating weight. It can represent the mean of the average gating weight.

[0107] Expert stability loss function: This loss function aims to mitigate the drastic oscillations in dynamic routing during the initial training phase. Based on the embedding feature representations of the current optimization epoch and the previous optimization epoch, the distance deviation between the embedding feature representations can be calculated. The mean of the sum of the squares of these distance deviations is used to construct the expert stability loss function. The expression can be:

[0108] in, It can represent the first During the second optimization process, the first Embedded feature representation of an expert, It can represent the first During the second optimization process, the first Embedded feature representation of an expert.

[0109] Based on the main task loss function, the expert load balancing loss function, and the expert stability loss function, a multi-objective loss function can be constructed. The expression can be:

[0110] in, This can represent the balance super loss of the expert load balancing loss function. This can represent the equilibrium hyperparameters of the expert stability loss function. It can be minimized using gradient-based optimization algorithms (such as Adam). Backpropagation updates the network parameters in the detection model. This includes parameters of the graph convolutional network, the projection matrix, and the routing network.

[0111] Furthermore, during operation, new users or items may be added. To ensure the continuity of the recommendation service, the following processing flow can be adopted without disrupting the existing optimization mechanism: (1) For newly added candidate items: When a newly added candidate item has multimodal content information, the feature information of multiple modalities of the newly added candidate item can be obtained by referring to the above description; the similarity with other candidate items is calculated based on the feature information of each modality; based on the similarity of each modality, the newly added candidate item is added to the semantic graph of the corresponding modality and corresponding edges are established. For example, the newly added candidate item is... Its content information includes image data and text data, and visual feature information is obtained by extracting features from it. and text feature information Retrieve Top-K similar candidate items from the semantic graph of the corresponding modality and establish connection edges. Incremental access to visual semantic graph and text semantic graph In one optional embodiment, to maintain the stability of the semantic graph freeze, the semantic graph can be updated in a batch / periodic manner, such as rebuilding the Top-K adjacency once a day or once a week, while the online inference stage only uses the most recently frozen semantic graph structure for propagation. Since the content expert network learns embedded feature representations on the semantic graph, even if a newly added candidate item has no interactive behavior, it can still rely on the semantic graph to obtain the usable representation of the item, thereby alleviating the cold start of the newly added candidate item.

[0112] (2) For new target users: After a new target user generates a small amount of interaction data such as browsing / clicking / purchasing, an edge relationship can be established between it and the candidate items that have already been interacted with. According to the denoising strategy, abnormal or unreliable interaction edges are filtered or downweighted, so that the new target user is incrementally connected to the denoised interaction graph. The structural expert network can propagate the target user node and candidate item nodes on the updated interaction graph, quickly forming the embedded feature representation of the target user; the routing network uses the interaction instances formed by the target user and candidate items. As input, the weight information of multiple expert networks is dynamically allocated, giving priority to more reliable signal sources. For example, when the content modalities are sufficient, the weight of the content expert network is biased, and as the interaction becomes richer, the weight of the structural expert network is gradually increased.

[0113] (3) For new target users without interactive behavior data: When the new target user has no historical interactive behavior data, the initial embedded features can be generated based on the target user's basic information or session / scene context information, and the prior routing allocation with content priority can be adopted. For example, the weight of visual / text experts can be increased and the weight of structural experts can be decreased. Available non-personalized or weakly personalized recommendations can be output first. As the target user's interactive behavior data is gradually accumulated, the routing mechanism can be used to achieve a smooth transition from prior routing guidance to data routing adaptation, thereby gradually enhancing the personalization effect.

[0114] Through the above-described incremental access and cold start processing procedures, the embodiments of this application can maintain recommendation availability in cases such as the addition of new target users, new candidate items, and missing modal information, and maintain consistency with the aforementioned expert network, routing, and denoising mechanisms.

[0115] To verify the effectiveness of the method proposed in this application, a comparative experiment was conducted on the publicly available first product category dataset Baby and the second product category dataset Clothing. The experimental results are shown in Table 1 below: Table 1

[0116] The comparative data shows that the complete method proposed in this application achieves optimal results on both types of datasets and metrics. A detailed analysis follows: Compared to variant A, the full method significantly improves recall and normalized depreciation cumulative gain, demonstrating that the progressive course routing strategy plays a crucial role in stabilizing the training of multimodal graph expert network architectures and preventing expert collapse, while direct adaptive learning often leads to suboptimal solutions.

[0117] Compared to variant B, the complete method significantly outperforms the original method, demonstrating that relying solely on user-item interaction data is insufficient. The successful extraction and utilization of rich visual and textual semantic information from items through the introduction of a multimodal graph expert architecture is of great significance for improving the accuracy and diversity of recommendations.

[0118] In summary, the embodiments of this application, by cleverly combining a multimodal graph expert network architecture and a progressive routing strategy, provide a new paradigm of multimodal recommendation that is high-performance, robust, and well-interpretable.

[0119] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0120] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0121] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also provides an information recommendation system.

[0122] refer to Figure 3 The information recommendation system 300 may include: The expert processing module 301 is used to calculate the embedding feature representation of the target user and each candidate item in the structural expert network based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items; to calculate the embedding feature representation of each candidate item in each content expert network based on the semantic graph constructed from the similarity relationship of the multiple candidate items in multiple modalities; and to associate each content expert network with the semantic graph of the corresponding modality.

[0123] The routing processing module 302 is used to process the interaction context features between the target user and each candidate item using the routing network to obtain a data routing distribution; and dynamically allocate the weight information of the prior routing distribution and the data routing distribution according to the time control factor to obtain a hybrid routing distribution; the time control factor decays as the optimization round increases.

[0124] The result prediction module 303 is used to perform gating selection on the hybrid routing distribution, determine the gating weights of each expert network; use the gating weights to process the embedded feature representations corresponding to the target user and each candidate item to obtain the user fusion representation and item fusion representation corresponding to each candidate item; and obtain the prediction score of each candidate item based on the user fusion representation and the item fusion representation to obtain the recommendation result.

[0125] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0126] The apparatus described above is used to implement the corresponding information recommendation method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the information recommendation method described in any of the above embodiments.

[0128] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0129] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0130] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0131] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0132] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0133] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0134] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0135] The electronic devices described above are used to implement the corresponding information recommendation methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0136] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the information recommendation method as described in any of the above embodiments.

[0137] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0138] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the information recommendation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0139] It should be noted that the embodiments of this application can also be further described in the following ways: An information recommendation method may include: calculating the embedding feature representations of the target user and each candidate item under a structural expert network based on an interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items; calculating the embedding feature representation of each candidate item under each content expert network based on a semantic graph constructed from the similarity of content data of multiple candidate items in multiple modalities; associating each content expert network with the semantic graph of the corresponding modality; processing the interaction context features between the target user and each candidate item using a routing network to obtain a data routing distribution; dynamically allocating the weight information of the prior routing distribution and the data routing distribution according to a time control factor to obtain a hybrid routing distribution; the time control factor decays with the increase of optimization rounds; performing gating selection on the hybrid routing distribution to determine the gating weights of each expert network; using the gating weights to process the embedding feature representations corresponding to the target user and each candidate item to obtain the user fusion representation and item fusion representation corresponding to each candidate item; and obtaining the prediction score of each candidate item based on the user fusion representation and the item fusion representation to obtain the recommendation result.

[0140] Optionally, based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, the embedding feature representations of the target user and each candidate item under the structural expert network are calculated; based on the semantic graph constructed from the similarity of the content data of multiple candidate items in multiple modalities, the embedding feature representation of each candidate item under each content expert network is calculated, including: obtaining the interaction graph based on the historical interaction behavior of the target user and candidate items; pruning the edges of the interaction graph based on the degree of the target user node and the degree of the candidate item node in the interaction graph to obtain a denoised interaction graph; extracting the feature information of each candidate item in multiple modalities based on the content data of multiple candidate items in multiple modalities; the feature information of multiple modalities is in the same mapping space; calculating the feature similarity of candidate items in each modality based on the feature information of multiple modalities to obtain the semantic graph of the corresponding modality; processing the denoised interaction graph and its corresponding initial embedding features using the structural expert network to obtain the embedding feature representations of the target user and multiple candidate items under the structural expert network; and processing the semantic graphs of the corresponding modalities and their corresponding initial embedding features using multiple content expert networks to obtain the embedding feature representations of multiple candidate items under multiple content expert networks.

[0141] Optionally, the routing network includes a first multilayer perceptron and a second multilayer perceptron. The routing network is used to process the interaction context features between the target user and each candidate item to obtain a data routing distribution. This includes: using the first multilayer perceptron to process the interaction context features between the target user and each candidate item to obtain a first selection preference score for each expert network; using the second multilayer perceptron to process the first selection preference score to obtain a second selection preference score for each expert network; and normalizing the second selection preference score to obtain the data routing distribution.

[0142] Optionally, based on a time control factor, the weight information of the prior route distribution and the data route distribution is dynamically allocated to obtain a hybrid route distribution, including: allocating weight information to the prior route distribution and the data route distribution according to the time control factor to obtain first weight information and second weight information; the first weight information is directly proportional to the time control factor, the second weight information is inversely proportional to the time control factor, and the time control factor decays as the optimization rounds increase; and the prior route distribution and the data route distribution are weighted and summed according to the first weight information and the second weight information to obtain the hybrid route distribution.

[0143] Optionally, gating selection is performed on the hybrid routing distribution to determine the gating weights of each expert network, including: injecting standard Gaussian noise into the hybrid routing distribution to obtain a perturbed hybrid routing distribution; selecting multiple expert networks that meet preset conditions from the perturbed hybrid routing distribution as candidate expert networks; normalizing the hybrid routing distribution of the candidate expert networks to obtain the corresponding gating weights; and setting the gating weights of the remaining expert networks to zero.

[0144] Optionally, by utilizing gating weights, the embedded feature representations of the target user and each candidate item are processed to obtain the user fusion representation and item fusion representation for each candidate item. This includes: for each candidate item, performing the following operations: obtaining the item fusion representation of the current candidate item based on the embedded feature representations of the current candidate item in the structural expert network and multiple content expert networks, as well as the corresponding gating weights; and obtaining the user fusion representation of the target user corresponding to the current candidate item based on the embedded feature representations of the target user in the structural expert network, the embedded feature representations of the current candidate item in multiple content expert networks, and the corresponding gating weights.

[0145] Optionally, the method further includes: updating the parameters of the expert network and the routing network with the objective of minimizing the multi-objective loss function; the multi-objective loss function is constructed based on the main task loss function, the expert load balancing loss function, and the expert stability loss function; the main task loss function is constructed based on the predicted scores of candidate items that the target user has interacted with and the predicted scores of candidate items that have not been interacted with; the expert load balancing loss function is constructed based on the gating weights of multiple candidate items on multiple expert networks; the expert stability loss function is constructed based on the embedding feature representation of the current optimization round and the embedding feature representation of the previous optimization round; and optimizing the recommendation results based on the updated parameters of the expert network and the routing network.

[0146] An information recommendation system may include: an expert processing module, used to calculate the embedding feature representations of the target user and each candidate item under a structural expert network based on an interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items; to calculate the embedding feature representation of each candidate item under each content expert network based on a semantic graph constructed from the similarity relationships of multiple candidate items under multiple modalities; and to associate each content expert network with the semantic graph of the corresponding modality; a routing processing module, used to process the interaction context features between the target user and each candidate item using a routing network to obtain a data routing distribution; to dynamically allocate the weight information of the prior routing distribution and the data routing distribution according to a time control factor to obtain a hybrid routing distribution; and to decrease the time control factor as the optimization rounds increase; and a result prediction module, used to perform gating selection on the hybrid routing distribution to determine the gating weights of each expert network; to process the embedding feature representations of the target user and each candidate item using the gating weights to obtain a user fusion representation and an item fusion representation corresponding to each candidate item; and to obtain a prediction score for each candidate item based on the user fusion representation and the item fusion representation to obtain a recommendation result.

[0147] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned information recommendation method.

[0148] A non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the aforementioned information recommendation method.

[0149] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0150] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0151] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0152] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An information recommendation method, characterized in that, include: Based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, the embedding feature representations of the target user and each candidate item under the structural expert network are calculated. Based on the semantic graph constructed from the similarity of the content data of the multiple candidate items in multiple modalities, the embedding feature representation of each candidate item under each content expert network is calculated; Each content expert network is associated with a semantic graph of the corresponding modality; The interaction context features between the target user and each candidate item are processed using a routing network to obtain the data routing distribution; Based on the time control factor, the weight information of the prior route distribution and the data route distribution is dynamically allocated to obtain the hybrid route distribution; the time control factor decreases as the optimization round increases; Gating selection is performed on the hybrid routing distribution to determine the gating weight of each expert network; Using the gating weights, the embedded feature representations of the target user and each candidate item are processed to obtain the user fusion representation and item fusion representation for each candidate item; Based on the user fusion representation and the item fusion representation, a predicted score is obtained for each candidate item to achieve a recommendation result.

2. The method according to claim 1, characterized in that, Based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items, the embedding feature representations of the target user and each candidate item under the structural expert network are calculated. Based on the semantic graph constructed from the similarity of the content data of the multiple candidate items across multiple modalities, the embedding feature representation of each candidate item under each content expert network is calculated, including: The interaction graph is obtained based on the historical interaction behavior between the target user and the candidate items; Based on the degree of the target user node and the degree of the candidate item node in the interaction graph, the edges of the interaction graph are pruned to obtain a denoised interaction graph. Based on the content data of the multiple candidate items in multiple modalities, feature information of each candidate item in multiple modalities is extracted; the feature information of the multiple modalities is in the same mapping space; Based on the feature information of multiple modalities, the feature similarity of candidate items within each modality is calculated to obtain the semantic map of the corresponding modality; The denoised interaction graph and its corresponding initial embedding features are processed using the structural expert network to obtain the embedding feature representations of the target user and the multiple candidate items under the structural expert network; By processing the semantic graphs of their corresponding modalities and their corresponding initial embedding features using the multiple content expert networks, the embedding feature representations of the multiple candidate items under the multiple content expert networks are obtained.

3. The method according to claim 1, characterized in that, The routing network includes a first multilayer perceptron and a second multilayer perceptron. The routing network processes the interaction context features between the target user and each candidate item to obtain a data routing distribution, including: The first multilayer perceptron and the interaction context features between the target user and each candidate item are processed to obtain the first selection preference score of each expert network; The first selection preference score is processed using the second multilayer perceptron to obtain the second selection preference score of each expert network; The second preference score is normalized to obtain the data routing distribution.

4. The method according to claim 1, characterized in that, Based on a time control factor, the weight information of the prior route distribution and the data route distribution is dynamically allocated to obtain a hybrid route distribution, including: Based on the time control factor, weight information is assigned to the prior route distribution and the data route distribution to obtain first weight information and second weight information; the first weight information is directly proportional to the time control factor, the second weight information is inversely proportional to the time control factor, and the time control factor decreases as the optimization rounds increase; Based on the first weight information and the second weight information, the prior route distribution and the data route distribution are weighted and summed to obtain a hybrid route distribution.

5. The method according to claim 1, characterized in that, Gating selection is performed on the hybrid routing distribution to determine the gating weights of each expert network, including: Standard Gaussian noise is injected into the hybrid route distribution to obtain a perturbed hybrid route distribution; Multiple expert networks that meet preset conditions are selected from the perturbed hybrid routing distribution as candidate expert networks; The hybrid routing distribution of the candidate expert networks is normalized to obtain the corresponding gating weights; the gating weights of the remaining expert networks are set to zero.

6. The method according to claim 1, characterized in that, Using the gating weights, the embedded feature representations corresponding to the target user and each candidate item are processed to obtain the user fusion representation and item fusion representation corresponding to each candidate item, including: For each candidate item, the operations performed include: Based on the embedding feature representations of the current candidate item in the structural expert network and the multiple content expert networks, as well as the corresponding gating weights, the item fusion representation of the current candidate item is obtained. Based on the embedding feature representation of the target user in the structural expert network, the embedding feature representation of the current candidate item in the multiple content expert networks, and the corresponding gating weights, the user fusion representation of the target user corresponding to the current candidate item is obtained.

7. The method according to claim 1, characterized in that, The method further includes: The parameters of the expert network and the routing network are updated with the goal of minimizing the multi-objective loss function. The multi-objective loss function is constructed based on the main task loss function, the expert load balancing loss function, and the expert stability loss function. The main task loss function is constructed based on the predicted scores of candidate items that the target user has interacted with and the predicted scores of candidate items that have not been interacted with. The expert load balancing loss function is constructed based on the gating weights of multiple candidate items on multiple expert networks. The expert stability loss function is constructed based on the embedding feature representation of the current optimization round and the embedding feature representation of the previous optimization round. The recommendation results are optimized based on the updated parameters of the expert network and the routing network.

8. An information recommendation system, characterized in that, include: The expert processing module is used to calculate the embedding feature representation of the target user and each candidate item in the structured expert network based on the interaction graph constructed from the historical interaction behavior of the target user and multiple candidate items. Based on the semantic graph constructed from the similarity relationships of the multiple candidate items in multiple modalities, the embedding feature representation of each candidate item in each content expert network is calculated; each content expert network is associated with the semantic graph of the corresponding modality. The routing processing module is used to process the interaction context features between the target user and each candidate item using a routing network to obtain a data routing distribution; and dynamically allocates the weight information of the prior routing distribution and the data routing distribution according to a time control factor to obtain a hybrid routing distribution; the time control factor decays as the optimization rounds increase; The result prediction module is used to perform gating selection on the hybrid routing distribution and determine the gating weights of each expert network; using the gating weights, the embedded feature representations corresponding to the target user and each candidate item are processed to obtain the user fusion representation and item fusion representation corresponding to each candidate item; based on the user fusion representation and the item fusion representation, the prediction score of each candidate item is obtained to obtain the recommendation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.

Images