A cross-domain multimodal recommendation method and system based on multi-agent attribute driving

Abstract

This application discloses a cross-domain multimodal recommendation method and system based on multi-agent attribute-driven architecture. Embodiments of this application can be applied to product recommendation scenarios in an e-commerce platform. The method includes: receiving multimodal features; processing the multimodal features using a first operation to generate reconstructed original multimodal features; the reconstructed original multimodal features include attribute-related representations and attribute-independent representations; then, processing the reconstructed original multimodal features using a second operation to generate a modal cue vector; then, processing the modal cue vector and the attribute-related representation to generate a new feature vector; and finally, processing the new feature vector using a sequence encoder to generate attribute-level multimodal sequence states. By sharing basic embeddings, the parameter overhead is significantly reduced, solving the problem that the parameter size of traditional cross-domain multimodal encoders increases linearly with the number of domains.

Description

Technical Field

[0001] This invention relates to the fields of recommendation systems and deep learning technology, and in particular to a cross-domain multimodal recommendation method and system based on multi-agent attribute-driven approach. Background Technology

[0002] With the rapid development of mobile internet and artificial intelligence technologies, recommendation systems have become a core link connecting users with massive amounts of information. Within the same e-commerce platform, with the explosive growth of multimodal content, single-modal feature representations are no longer sufficient to meet users' needs for accurate recommendations; therefore, cross-domain multimodal recommendation systems have emerged.

[0003] Meanwhile, with the continuous diversification of consumer demands, the product categories on e-commerce platforms are constantly being enriched and iterated at an increasingly rapid pace. The range of products now covers all aspects of people's lives, including 3C digital products, fresh produce, baby and children's toys, books and audio-visual products, clothing and footwear, home appliances, and beauty and skincare products. Furthermore, within each sub-sector (i.e., each domain) of the platform, products contain rich multimodal features, such as textual descriptions (titles, detailed descriptions, user reviews), visual information (main product images, detailed images, promotional video frames), and audio content (product explanation audio, review voice recordings). These multimodal features collectively constitute a complete description of product attributes and are the core data support for achieving accurate recommendations. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a cross-domain multimodal recommendation method and system based on multi-agent attribute-driven approaches; According to one aspect, a cross-domain multimodal recommendation method based on multi-agent attribute-driven approach is provided, comprising: receiving multimodal features; processing the multimodal features using a first operation to generate reconstructed original multimodal features; the reconstructed original multimodal features including attribute-related representations and attribute-independent representations; then, processing the reconstructed original multimodal features using a second operation to generate a modality cue vector; then, processing the modality cue vector and the attribute-related representation to generate a new feature vector; and processing the new feature vector using a sequence encoder to generate attribute-level multimodal sequence states.

[0005] The above technical solution has the following advantages or beneficial effects: Traditional methods directly use sequence encoders to encode multimodal features. However, in cross-domain scenarios, the distribution of modal features differs significantly across domains, making it difficult to adapt the same encoder to features from multiple domains simultaneously, resulting in poor cross-domain knowledge transfer. To address this, this embodiment, after obtaining the user's attribute-modal dual-dimensional representation, first generates modal cue vectors through modal cue learning. These cue vectors are then input into the sequence encoder to generate attribute-level multimodal sequence states. Specifically, a basic encoder shared across all domains is used to initially encode the products, obtaining a basic embedding vector. Then, a cue vector is designed for each modality in each domain, and this cue vector is added to the basic embedding vector to obtain the modal cue vector. By fusing modal cue learning technology and sharing the basic embedding, parameter overhead is significantly reduced, fundamentally solving the problem of the linear growth of parameter size with the number of domains in traditional cross-domain multimodal encoders, thereby achieving more accurate recommendations.

[0006] Compared with traditional methods, the embodiments of this application have the following advantages: First, fine-grained attribute-level knowledge transfer is achieved through attribute-driven decoupling technology, effectively filtering source domain-irrelevant features, significantly reducing negative transfer phenomena, and improving recommendation accuracy. Second, a multi-agent reinforcement learning framework is introduced, in which each attribute agent dynamically selects cross-domain knowledge matching the target domain, adapting to the evolution of user interests in real time and overcoming the limitations of traditional static selection. Furthermore, attribute-supervised VAEs separate multimodal features into attribute-related and attribute-irrelevant representations, improving the interpretability of recommendation results. Finally, innovative modal cue learning technology is integrated, significantly reducing the parameter size through shared basic embeddings, solving the problem of linear parameter growth with the number of domains in traditional methods, reducing training and inference costs, and facilitating practical deployment. Attached Figure Description

[0007] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0008] Figure 1 This is an example workflow for the recommended method in Implementation Example 1; Figure 2 A flowchart illustrating the recommended method in Example 1 Figure 1 ; Figure 3 A flowchart illustrating the recommended method in Example 1 Figure 2 ; Figure 4 This is a flowchart illustrating the recommendation system of Example 2. Detailed Implementation

[0009] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0010] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0011] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of this invention, "multiple" refers to two or more.

[0012] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0013] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0014] Example 1 In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0015] All data is derived from publicly available Amazon datasets and is obtained and used in a reasonable and legal manner.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] 1) Cross-domain multimodal recommendation refers to recommendation methods that leverage cross-domain knowledge and multimodal information (text, visual, audio, etc.) from multiple domains to improve recommendation performance. Specifically, cross-domain recommendation involves transferring knowledge from the source domain to the target domain to alleviate the data sparsity problem in the target domain; multimodal recommendation utilizes multiple modal features such as text descriptions, images, and audio for item recommendations. Cross-domain multimodal recommendation combines the advantages of both, utilizing cross-domain knowledge transfer and multimodal feature fusion to achieve more accurate recommendations.

[0018] 2) A source domain refers to one or more domains with relatively abundant data and existing user-item interaction data, used to provide transferable knowledge. For example, in the Amazon dataset experiment, the film and television and toy / game domains can be used as source domains.

[0019] 3) The target domain refers to the domain that requires recommendations and may have data sparsity issues; it is the recipient of knowledge transfer. For example, in the Amazon dataset experiment, the book domain can be the target domain.

[0020] 4) An item refers to a recommended object in a recommendation system, namely, an entity that can be recommended, such as a product, movie, or book. Each item contains multiple attributes and multimodal features and is the basic unit of user interaction.

[0021] 5) Modal features refer to the way an item is presented in different modalities, including but not limited to: text modality, visual modality, audio modality, etc. These multimodal features together constitute a complete description of the item and are the core data support for achieving accurate recommendations.

[0022] Traditional cross-domain multimodal recommendation methods involve designing an independent modal encoder for each modality within each domain. Specifically, within the same e-commerce platform, a dedicated 3C visual encoder is designed for the visual features of 3C products; a separate fresh produce visual encoder is designed for the visual features of fresh produce; a dedicated toy text encoder is built for the text features of toy products; a separate book text encoder is constructed for the text features of book products, and so on. This design scheme results in a strict linear positive correlation between the number of encoders, the size of the model parameters, and the number of domains within the e-commerce platform.

[0023] In the long run, as the platform continues to develop and new sub-fields are added, the number of corresponding dedicated encoders will surge, and the number of model parameters will grow exponentially, leading to a series of prominent problems: On the one hand, a large number of redundant parameters will significantly increase the training difficulty and cost of the model, making it difficult to deploy in actual systems; on the other hand, the surge in the number of parameters will significantly reduce the inference speed of the recommendation system, resulting in delayed response to recommendation results. At the same time, too many dedicated encoders are prone to problems such as feature redundancy and feature conflicts, which will significantly reduce the recommendation accuracy of the recommendation system, making it unable to accurately match the user's real shopping needs. Ultimately, this will lead to increased time spent by users searching for products, a worse shopping experience, and even user churn, making it difficult to adapt to the large-scale and refined development needs of e-commerce platforms.

[0024] Furthermore, traditional cross-domain recommendation methods generally employ a coarse-grained, full-item transfer strategy, transferring all knowledge of products from the source domain to the target domain as a whole. This includes common useful knowledge, domain-specific irrelevant knowledge, and redundant interference knowledge. This indiscriminate transfer of knowledge leads to high noise levels in cross-domain transfer, easily introducing irrelevant knowledge and causing negative transfer, severely impacting recommendation accuracy. Traditional methods rely on static similarity calculations for cross-domain selection, typically pre-calculating fixed inter-domain similarities based on historical statistical data of the source and target domains to determine the scope and weight of knowledge transfer. This calculation method is static and cannot adapt to dynamic changes in user interests. In e-commerce scenarios, user shopping interests often dynamically adjust with time, context, and needs. Traditional static similarity calculations cannot capture these dynamic changes in real time and continue to perform cross-domain knowledge transfer according to a fixed strategy, resulting in a mismatch between transferred knowledge and real-time user interests, further reducing recommendation accuracy. Moreover, source domain products contain multiple attribute dimensions, and the selection strategies for different attribute dimensions often conflict, lacking an effective coordination mechanism. Furthermore, traditional multimodal recommendation methods directly encode textual, visual, and detail features globally, resulting in a severe entanglement between multimodal features and product attributes. This makes it difficult to distinguish between different attribute dimensions such as brand, price, and style, leading to a lack of interpretability in the recommendation results.

[0025] In cross-domain multimodal recommendation scenarios, items (items, i.e., recommended entities) in the source and target domains typically contain multiple modal features such as text, visual, and audio. However, traditional multimodal coding methods fuse all modal features into a unified representation, which presents two key problems: (1) Features from different modalities are often intertwined, making it difficult to distinguish which features are related to user decision-making attributes and which are noise information; (2) During cross-domain transfer, a large number of features from the source domain that are unrelated to the target domain introduce noise, leading to negative transfer and severely reducing recommendation accuracy.

[0026] This embodiment first constructs a cross-domain dataset, selecting three domains from the Amazon dataset: film and television, toys and games, and books. Within these three domains, one domain is chosen as the target domain, and the other two as the source domains, thus forming three different cross-domain recommendation tasks. For the data in these three domains, users and items with fewer than five interactions are filtered out. Then, users with overlapping cross-domain interactions are selected to construct the final cross-domain dataset.

[0027] Further, see Figure 1-3 , Figure 1 An example workflow100 for a cross-domain multimodal recommendation method driven by multi-agent attributes is described. Figure 2 This is a flowchart illustrating the recommended method provided in Implementation Example 1. Figure 1 ; Figure 3 A flowchart illustrating the recommended method in Example 1 Figure 2 . will combine Figure 1-3 The steps shown are explained.

[0028] S1. Perform attribute-level decoupling on the multimodal features of the source and target domains to obtain a two-dimensional representation of the project's attributes and modalities.

[0029] See Figure 1 As shown, the workflow 100 in this embodiment begins with the input of multimodal features. In some examples, multimodal features It can be a variety of modal features, such as text features, visual features, or audio features.

[0030] In some respects, multimodal features The extraction process is as follows: for text modalities, a pre-trained language model (e.g., BERT) is used to encode the text description into a vector representation; for visual modalities, a convolutional neural network (e.g., ResNet) is used to extract image features; for audio modalities, an acoustic model (e.g., Wav2Vec) is used to extract audio features. Each modality encoder outputs a feature vector for that modality.

[0031] In cross-domain recommendation tasks that transfer knowledge from the film and television, toy and game, and book domains, multimodal features are crucial. The integration includes two categories: source domain multimodal features and target domain multimodal features. Source domain multimodal features extract features from products in the film and television, toy, and game domains, providing transferable knowledge; examples include visual features of movie posters and textual features of movie titles. Target domain multimodal features extract features from products in the book domain and are the primary objects of recommendation; examples include visual features of book covers and textual features of book titles. Both encompass textual and visual modalities but originate from different domains.

[0032] Furthermore, multimodal features By using attribute-supervised variational autoencoders (VAEs), attribute-dependent and attribute-independent representations are obtained, achieving attribute-modal decoupling in two dimensions.

[0033] Design an attribute-supervised variational autoencoder to decompose the encoded multimodal features into two parts: attribute-dependent representation and attribute-independent representation. The decoupling process is achieved through the following formula:

[0034] in, , , These represent the original encoded features of text, visual, and audio modalities, respectively. For attribute-related representation, This is an attribute-independent representation. Attribute-related representations contain key attribute information relevant to user decisions, such as book type, author, price, and rating; movie type, actors, and year; and product brand and category. Attribute-independent representations, on the other hand, include background noise and irrelevant details, such as cover background color, font style, poster watermark, and camera angle.

[0035] Advantageously, attribute-supervised variational autoencoders can map features from different domains and modalities (such as visual features from the 3C domain and text features from the fresh food domain) to the same latent space, generating latent features of a unified dimension. This can effectively solve the problems of parameter redundancy and the inability to share cross-domain features in traditional modal encoders.

[0036] Furthermore, after encoding with an attribute-supervised variational autoencoder, the original multimodal features are reconstructed. Attribute-related representation Independent of attributes splicing and reconstructing the original multimodal features This concatenation operation is used to combine attribute-related representations. Independent of attributes The feature dimensions are connected end to end.

[0037] Reconstructing the original multimodal features It can be expressed as:

[0038] in, For attribute-related representation, This is an attribute-independent representation. This is for splicing operations.

[0039] Advantageously, reconstructing the original multimodal features Z enables explicit feature decoupling, separating attribute-related and attribute-irrelevant parts for individual use. During transfer, only attribute-related representations can be transferred to avoid noisy negative transfer, improving cross-domain transfer performance; attribute-related parts can be mapped to specific product attributes, thereby enhancing interpretability; feature compression and denoising can also be achieved, reducing computational overhead; and feature representations from different domains and modalities can be unified into the same latent space. The reconstructed original multimodal features Z are themselves attribute-modality two-dimensional representations, internally organized according to attribute and modality dimensions, representing a structurally decoupled feature representation. The reconstructed original multimodal features Z are input into modality cue learning to generate cue vectors, cross-attention mechanisms achieve inter-domain feature alignment, and similarity calculation, spanning the entire process from encoding to prediction.

[0040] S2. Based on the interaction sequences of users in the target domain, generate attribute-level multimodal sequence states of users through modal cue learning; Traditional methods directly encode multimodal features using sequence encoders. However, in cross-domain scenarios, the distribution of modal features differs significantly across domains, making it difficult to adapt the same encoder to features from multiple domains simultaneously, resulting in poor cross-domain knowledge transfer. To address this, this embodiment, after obtaining the user's attribute-modality dual-dimensional representation (i.e., reconstructing the original multimodal features Z), first generates modal cue vectors through modal cue learning. These cue vectors are then input into the sequence encoder to generate attribute-level multimodal sequence states. Specifically, a basic encoder shared across all domains is used to initially encode the product, obtaining a basic embedding vector. Then, a cue vector is designed for each modality in each domain, and this cue vector is added to the basic embedding vector to obtain the modal cue vector. By fusing modal cue learning technology and sharing the basic embedding, parameter overhead is significantly reduced, fundamentally solving the problem of the linear growth of parameter size with the number of domains in traditional cross-domain multimodal encoders.

[0041] In some respects, the interaction sequence items for target domain users are extracted from the user's historical behavioral data on the target platform. The system records all user actions in the target domain, including purchases, browsing, favorites, ratings, etc., and then arranges them in chronological order to form a time-sorted product list. For example, a user's history on the Amazon Books platform might show: purchased "The Three-Body Problem" on January 1st, browsed "The Wandering Earth" on January 3rd, favorited "Folding Beijing" on January 5th, and purchased "Foundation" on January 10th. Arranging these products in chronological order yields the user's interaction sequence items.

[0042] In other words, the interaction sequence items for target domain users refer to the set of items that target domain users have operated on in chronological order during their historical interactions. The system extracts these item records from the user behavior database, sorts them by timestamp, and forms an interaction sequence. Then, each item in the sequence is processed, combining the multimodal features of the item with a modality-specific cue vector to generate an enhanced modality cue vector for each item, which is used for subsequent sequence modeling and user preference modeling.

[0043] Furthermore, modal cueing learning divides model parameters into two parts: fixed shared parameters and slowly growing cue parameters. Shared parameters refer to the parameters of the base encoder, including the text encoder, image encoder, projection layers, etc. These parameters are numerous, typically ranging from tens of millions to hundreds of millions. However, they are shared across all domains and do not increase with the number of domains. Whether the system processes one domain, ten domains, or one hundred domains, the number of these parameters remains constant.

[0044] Cue parameters refer to the cue vectors for each domain and each modality. These vectors have a very small number of parameters, typically only one-tenth or even less of the number of parameters in the base encoder. Their number increases with the number of domains, but the overall growth is very slow due to the small number of parameters in a single vector. In actual operation, the system selects the corresponding cue vector and combines it with the shared base embedding based on the domain being processed. In this way, the system can utilize the general multimodal representation capabilities learned by the shared encoder while also adapting to the specific characteristics of different domains through the cue vectors.

[0045] Modal cue learning is performed on the interaction sequence items of the target domain user to generate a modal cue vector for each item. Modal cue vector The generation process can be expressed as:

[0046] in, This is the initial embedding of the project, i.e., the basic embedding vector; For modality The prompt vector; i refers to the position number of the product in the user interaction sequence, which indicates which product in the user interaction sequence, and m refers to the modality type number, which indicates which modality.

[0047] Furthermore, the initial embedding of the project It is obtained through a shared pre-trained encoding network, that is, by sharing the encoding of the reconstructed original multimodal features Z. For text, the system uses a pre-trained language model (such as Google's BERT model) to convert textual information such as product titles, descriptions, and user reviews into numerical vectors. For images, the system uses a pre-trained convolutional neural network (such as Google's ResNet model) to convert product cover images into numerical vectors. For audio (if any), the system uses a pre-trained speech model to convert audio files into numerical vectors. All these vectors are adjusted in dimension and then merged into a unified basic embedding vector.

[0048] Modal cue vector These are the learnable parameters of the model; they are not fixed in advance but are gradually learned during model training. At the start of model training, these cue vectors are randomly initialized or use some pre-trained vectors as initial values. Then, during the training of the recommendation task, the system automatically adjusts the values ​​of these cue vectors based on the recommendation performance using a backpropagation algorithm. After extensive training, the cue vectors automatically learn the specific features and patterns of that modality on that platform.

[0049] Furthermore, after generating modal cue vectors through modal cue learning, these vectors are then input into a sequence encoder to generate attribute-level multimodal sequence states; specifically, the m-th modal cue vector of the i-th item is... Representation of attributes related to the i-th item Element-wise product is performed to obtain a new feature vector; subsequently, the new feature vector is input into the Transformer sequence encoder, which generates attribute-level multimodal sequence states. .

[0050] Attribute-level multimodal sequence state The generation process can be represented as:

[0051] in, For the project The attribute-related representation, Let i be the modal cue vector. For element-wise product, For target domain users The interaction sequence. By fusing multimodal cue learning and sharing basic embeddings, the parameter overhead is significantly reduced, solving the problem that the parameter size of traditional cross-domain multimodal encoders increases linearly with the number of domains.

[0052] It's important to note that traditional methods for handling cross-domain multimodal recommendation typically input the original multimodal features directly into the encoder, or train a separate encoder for each domain. This leads to a linear increase in the number of encoders needed as the number of domains increases, resulting in an explosive growth in the total number of parameters. This embodiment addresses this by first generating modal cue vectors. First, a base encoder shared across all domains is used to initially encode the items, obtaining a base embedding vector. Then, a cue vector is designed for each modality in each domain, and this cue vector is added to the base embedding vector to obtain the modal cue vector. This way, even if the base encoder has a large number of parameters, it is shared across all domains, so this parameter count does not increase regardless of the number of domains. Only the cue vector increases, and the number of parameters in the cue vector is very small, resulting in a very slow increase in the total number of parameters even with a large number of domains.

[0053] S3. Design an agent for each source domain attribute. The agent observes the state of the attribute-level multimodal sequence and dynamically selects the source domain multimodal item that matches the target domain attribute.

[0054] In cross-domain scenarios, source domain items often contain a large number of noisy features that are unrelated to the target domain. Direct transfer will lead to negative transfer and reduce recommendation accuracy. In addition, in multi-source domain scenarios, the knowledge transfer strategies of different source domains are independent of each other and lack a global coordination mechanism, which may lead to cross-domain knowledge conflicts and make it impossible to form a unified recommendation strategy.

[0055] In this embodiment, a multi-agent reinforcement learning framework is introduced to achieve accurate knowledge selection at the attribute level. A multi-agent collaborative optimization algorithm, namely the hybrid value network algorithm (QMIX algorithm), is introduced to achieve collaborative decision-making among multiple agents, ensuring the globally optimal recommendation strategy.

[0056] In this context, multi-agent systems refer to multiple independent decision-making units, each with its own independent observation, decision-making, and learning capabilities. Unlike traditional reinforcement learning where there is only one agent, multi-agent systems involve multiple agents sharing the same environment but with different goals or behavioral strategies. In this embodiment, the multi-agent system is designed as follows: First, the system designs an independent agent for each attribute of each source domain. Assuming there are three source domains (movie domain, book domain, and toy domain) and four attributes (brand, category, price, and style), the system will create twelve agents: a brand agent for the movie domain, a category agent for the movie domain, a price agent for the movie domain, a style agent for the movie domain, and corresponding attribute agents for the book and toy domains. Each agent has its own specific responsibility. For example, the brand agent for the movie domain is responsible for observing the target domain users' brand preferences and then selecting the product from the movie domain that best matches their brand preferences. The category agent for the book domain is responsible for observing the target domain users' category preferences and then selecting the product from the book domain that best matches their category preferences. This fine-grained design allows each agent to focus on only one specific dimension, without considering other dimensions, thus enabling more accurate decisions. Each agent operates independently without interfering with others, but their choices collectively influence the final recommendation results.

[0057] The multi-agent cooperative optimization algorithm, namely the QMIX algorithm, adopts a centralized training and distributed execution architecture. During the training phase, the system considers the joint actions of all agents and calculates the globally optimal policy, but each agent makes independent decisions during actual execution.

[0058] An agent is designed for each source domain attribute. This agent observes the state of the attribute-level multimodal sequence and dynamically selects source domain multimodal items that match the target domain attribute. This includes designing an independent agent for each attribute of each source domain. For example, in a scenario with N source domains and K attributes, a total of N×K agents are designed. Each agent is responsible for observing the target domain user's preference state on that attribute and selecting the most relevant item from the corresponding source domain. This fine-grained agent design ensures the accuracy of knowledge transfer, transferring only truly relevant knowledge for each attribute.

[0059] In workflow 100, multi-agent selection is used to select attribute-level multimodal sequence states. The process yields attribute-level source domain candidates G.

[0060] Each attribute Corresponding to an intelligent agent The observation space of the intelligent agent is:

[0061] in, For target domain users The Attribute sequence state, For the source domain A set of candidate attributes.

[0062] The agent's action space is a distribution of candidate selection probabilities, and the selection probabilities of the candidates are output through a deep Q-network (DQN network). ,in This is the observation vector.

[0063] DQN networks are reinforcement learning algorithms that combine deep learning and Q-learning. Q-learning is a classic reinforcement learning method that learns a value function to evaluate the merits of taking each action in each state. The Q-value is a value evaluation metric that represents the expected long-term cumulative reward for taking an action in a given state. Simply put, the higher the Q-value, the more worthwhile the action is. In this embodiment, the Q-value of each agent represents the long-term contribution of selecting a candidate option from the source domain to the recommendation effect, given the current preference state of the user in the target domain. The higher the Q-value, the more helpful the selection of this candidate option is for the recommendation in the target domain.

[0064] In this embodiment, the Q-values ​​of each agent are calculated using their respective DQN networks. Specifically: First, the agent acquires an observation vector. This observation vector contains the target domain user's preference state on the corresponding attribute, as well as candidate information for that attribute in the source domain. Then, the agent inputs the observation vector into its own DQN network. The DQN network, through multi-layer neural network calculations, outputs the Q-value for each candidate. If there are ten candidate options, the DQN network will output ten Q-values. These Q-values ​​reflect the agent's evaluation of the value of each candidate option. Candidates with high Q-values ​​are considered by the agent to be more helpful for recommending the target domain; candidates with low Q-values ​​are considered by the agent to be less helpful for recommending the target domain.

[0065] Furthermore, the Q-values ​​of each agent are fused into a global Q-value using the QMIX algorithm. The fusion process can be represented as follows: , in, As attribute weights, multi-agent cooperative strategies are optimized through centralized training; It is the local Q-value of the k-th attribute agent.

[0066] Advantageously, by introducing a multi-agent dynamic selection mechanism, an independent agent is designed for each source domain attribute, enabling attribute-level accurate knowledge selection and effectively reducing cross-domain noise and negative transfer.

[0067] S4. By aligning the attribute-modal representations of selected items in the source domain with the attribute-modal representations of items in the target domain through a cross-attention mechanism, a domain-shared attribute-level multimodal sequence representation is generated.

[0068] After selecting attribute-level source domain candidates G using multi-agent methods, it is necessary to effectively fuse this cross-domain knowledge with the knowledge of the target domain. However, there are significant distributional differences between the attribute-modal representations of the source and target domains, making direct concatenation or simple weighted fusion difficult to achieve effective knowledge transfer. Traditional feature alignment methods are usually based on statistical matching or adversarial learning, but these methods require a large number of alignment samples and struggle to handle the complex differences between the attribute-modal dimensions. Achieving accurate alignment of inter-domain representations and generating domain-shared attribute-level multimodal sequence representations is a pressing technical challenge.

[0069] This embodiment introduces a cross-attention mechanism, which actively queries the source domain representation through the target domain representation to achieve accurate inter-domain feature alignment.

[0070] In workflow 100, the target domain item is represented by its attribute-modal structure (i.e., attribute-level multimodal sequence state). For the query, the attribute-modal representation of the selected item in the source domain (i.e., the attribute-level source domain candidate G) is a key-value pair. Cross-attention weights are calculated as follows:

[0071] in, For target domain items Attributes - Modal query vectors, Select item for source domain Attributes - Modal Key Vector For vector dimensions.

[0072] Then, the calculated cross-attention weights are used. We perform a weighted summation of the source domain value vectors to obtain the cross-domain aligned representation:

[0073] in, Select item for source domain The attribute - modal value vector.

[0074] Employing a cross-attention mechanism enables the alignment of attribute-modal representations across domains, generating domain-shared multimodal sequence representations.

[0075] S5. Integrate the domain-shared representation with the target domain sequence representation to predict the user's next interaction item.

[0076] In the prediction phase, a multi-layered representation fusion mechanism is designed to deeply fuse the target domain sequence representation with the cross-domain aligned representation, predicting the user's next interaction item through similarity calculation. Simultaneously, a joint optimization strategy is used to optimize the recommendation task, attribute decoupling task, and multi-agent selection task, achieving end-to-end model training.

[0077] The multi-level representation fusion mechanism in this embodiment is a technical solution that deeply fuses the target domain and the cross-domain aligned representations at different levels during the prediction phase. This mechanism includes two levels of fusion: First layer: Attribute-level fusion. For each attribute, the attribute-level multimodal sequence state is fused. Cross-domain alignment representation Perform splicing. Attribute-level multimodal sequence states. It is the original attribute representation of the target domain, reflecting the inherent preferences of users in the target domain on that attribute; cross-domain alignment representation It is a cross-domain aligned attribute representation that incorporates cross-domain knowledge from the source domain. Through concatenation, the two representations are combined, preserving their complete information and providing rich features for subsequent recommendation prediction.

[0078] The second layer: Cross-attribute fusion. The fused representations of all attributes are performed across attributes using an attention mechanism to obtain the final user representation. The attention pooling layer automatically learns the importance of different attributes and adaptively adjusts the weights of different attributes. This mechanism can highlight the attribute features that users truly care about and suppress unimportant attributes.

[0079] The next interaction item refers to the item that the system predicts users are most likely to interact with in the future. In the context of recommendation systems, interaction can refer to various user behaviors, including purchasing, browsing, saving, and rating items. The next interaction item is the object on which the system predicts the user's most likely next action.

[0080] Specifically, in workflow 100, the attribute-level multimodal sequence state is... Cross-domain alignment representation By combining the data, the final user representation can be obtained. Subsequently, the user's statement was calculated. Similarity to candidate representations, specifically: using a softmax operation to compute the final user representation. Similarity to candidate item representations. Candidate item representations refer to the feature representations of all potentially recommended items in the system, including items from both the target and source domains. The system needs to calculate the similarity between the user's representation and these candidate items, and select the most similar item as the recommendation result.

[0081] Calculate the final user representation The similarity to the candidate options can be represented as:

[0082] in, It is the final user representation. Reconstructing the original multimodal features of item i.

[0083] Finally, the cross-entropy loss of the recommendation task is used. Classification loss due to attribute decoupling and multi-agent reinforcement learning loss Joint optimization model:

[0084] in, , These are the weight coefficients for attribute decoupling classification loss and multi-agent reinforcement learning loss, respectively.

[0085] Cross-entropy loss for recommendation tasks To ensure information integrity and avoid information loss due to excessive decoupling, the formula is as follows:

[0086] Where p is used to calculate the similarity between users and candidates; i represents possible candidates; It is the final user representation.

[0087] Classification loss due to attribute decoupling To ensure that the attribute-related representations decoupled from multimodal features have good attribute discrimination ability, the formula is as follows:

[0088] in, It is the item's true attribute label (one-hot encoded vector). It is a representation of the item's attributes (obtained from decoupling from VAE). It is an attribute classifier composed of a fully connected neural network.

[0089] Multi-agent reinforcement learning loss The formula for guiding an agent to learn the most valuable cross-domain knowledge selection strategy is as follows:

[0090] in, It is the observation vector of the i-th sample. It is the joint action vector of all agents in the i-th sample. Each sample corresponds to one interaction between the user and the recommendation system. It is the Q value after QMIX fusion. The target Q value for the i-th sample is calculated using the following formula:

[0091] in, It is the immediate reward obtained by the i-th sample. It's a discount factor, and its importance lies in controlling future rewards. It is the Q value of the target network. It is the observation vector of the next state. It represents all possible action combinations for the next state.

[0092] Example 2 This embodiment provides a cross-domain multimodal recommendation system based on multi-agent attribute driving; Figure 4 The recommendation system 200 of Embodiment 2 can be configured to perform aspects of the methods described herein.

[0093] Recommender system 200 includes a data processing module 201, which selects three domains from the Amazon dataset: film and television, toys and games, and books. Within these three domains, one domain is designated as the target domain, and the other two as the source domains, thus forming three different cross-domain recommendation tasks. For the data in these three domains, users and items with fewer than five interactions are filtered out, and then users with overlapping cross-domain interactions are selected to construct the final cross-domain dataset. The data processed by this module is then input into the multimodal attribute decoupling module 202.

[0094] The multimodal attribute decoupling module 202 encodes the multimodal features of an item, achieving attribute-modal two-dimensional decoupling through an attribute-supervised variational autoencoder (VAE), obtaining attribute-related and attribute-independent representations. These representations are then concatenated to obtain the reconstructed original multimodal features. The reconstructed original multimodal features are then input into the modality cueing learning module 203.

[0095] The modal cue learning module 203 first generates a modal cue vector for each item, and then generates attribute-level multimodal sequence states through a sequence encoder. The attribute-level multimodal sequence states are then input into the multi-agent reinforcement learning module 204.

[0096] The multi-agent reinforcement learning module 204 uses a multi-agent reinforcement learning model and a multi-agent collaborative optimization algorithm to select candidate sources for the source domain. The candidate sources are then input into the fusion module 205.

[0097] The fusion module 205 uses the attribute-level multimodal sequence state as the query and the attribute-level source domain candidates as key-value pairs. It calculates the cross-attention weights and performs a weighted summation of the source domain value vectors to obtain the cross-domain aligned representation. The cross-domain aligned representation is then input into the prediction module 206.

[0098] The prediction module 206 concatenates the target domain sequence representation with the cross-domain aligned representation to obtain the final user representation. It then calculates the similarity between the user representation and the candidate representation to predict the user's next interaction item.

[0099] Finally, the model is jointly optimized through the task optimization module 207, which includes the multimodal attribute decoupling module 202, the multi-agent reinforcement learning module 204, and the prediction module 206, respectively using attribute decoupling classification loss, multi-agent reinforcement learning loss, and cross-entropy loss for recommendation tasks for optimization.

[0100] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should also be noted that the above modules, as part of a system, can be executed in a computer system such as a set of computer-executable instructions.

[0101] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0102] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0103] Example 3 This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0104] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0105] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0106] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0107] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0108] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0109] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-domain multimodal recommendation method based on multi-agent attribute-driven approach, characterized in that, include: Receive multimodal features; The multimodal features are processed using a first operation to generate reconstructed original multimodal features; the reconstructed original multimodal features include attribute-related representations and attribute-independent representations; First, the second operation is used to process the reconstructed original multimodal features to generate a modal cue vector; then, the modal cue vector and the attribute-related representation are processed to generate a new feature vector; finally, a sequence encoder is used to process the new feature vector to generate an attribute-level multimodal sequence state.

2. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 1, characterized in that, The first operation includes an attribute-supervised variational autoencoder and a concatenation operation; the concatenation operation is used to connect the feature dimensions of attribute-related representations and attribute-independent representations end to end; The second operation includes modal cue learning, which generates a modal cue vector for each item; the modal cue vector for each item is obtained by adding the cue vector for each modality to the basic embedding vector.

3. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 1, characterized in that, The attribute-level multimodal sequence state is generated by first creating a new feature vector from the m-th modal cue vector of the i-th item and the attribute-related representation of the i-th item; then, the new feature vector is processed by a sequence encoder to generate the attribute-level multimodal sequence state.

4. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 3, characterized in that, The attribute-level multimodal sequence states are processed by a multi-agent reinforcement learning model to obtain attribute-level source domain candidates. The multi-agent reinforcement learning model includes a multi-agent reinforcement learning framework and a multi-agent collaborative optimization algorithm.

5. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 4, characterized in that, It also includes using attribute-level multimodal sequence states as queries and attribute-level source domain candidates as key-value pairs to calculate cross-attention weights; and using the calculated cross-attention weights to perform a weighted summation of the source domain value vectors to obtain a cross-domain aligned representation.

6. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 1, characterized in that, The attribute-level multimodal sequence state is concatenated with the cross-domain aligned representation to obtain the final user representation; the user's next interaction item is predicted by calculating the similarity between the final user representation and the candidate representation.

7. The cross-domain multimodal recommendation method based on multi-agent attribute driving as described in claim 1, characterized in that, Cross-entropy loss for recommendation tasks Classification loss due to attribute decoupling and multi-agent reinforcement learning loss Joint optimization recommendation task, multimodal attribute decoupling task, and multi-agent attribute-level source domain item selection task.

8. A cross-domain multimodal recommendation system based on multi-agent attribute-driven architecture, characterized in that, include: A multimodal attribute decoupling module is configured to receive multimodal features and process the multimodal features using a first operation to generate reconstructed original multimodal features; the reconstructed original multimodal features include attribute-related representations and attribute-independent representations; The modal cue learning module is configured to first use a second operation to process the reconstructed original multimodal features to generate a modal cue vector; then, process the modal cue vector and the attribute-related representation to generate a new feature vector; and finally, use a sequence encoder to process the new feature vector to generate an attribute-level multimodal sequence state.

9. An electronic device, characterized in that, include: A memory for non-transitory storage of computer-readable instructions; and a processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 7.

Images