Multi-view sequence recommendation method and system based on multi-modal fine-grained information

Abstract

The application discloses a multi-view sequence recommendation method and system based on multi-modal fine-grained information, and relates to the technical field of personalized recommendation, comprising: obtaining historical commodity interaction data and commodity metadata of a user, constructing a knowledge graph, and obtaining commodity embedding representation; obtaining text modal data and image modal data related to commodities, respectively performing fine-granularity, and generating corresponding fine-grained feature embedding representation; fusing the commodity embedding representation and the fine-grained feature embedding representation of different modalities, and optimizing fine-grained fusion by using a contrast learning strategy to obtain comprehensive commodity embedding representation mapping to a multi-head hidden space, modeling user preferences from different fine-grained perspectives, aggregating the recommendation probability of candidate commodities from each fine-grained perspective, obtaining the final recommendation probability, and predicting the next commodity demand. The application performs personalized recommendation through the multi-head hidden space, reveals the interest degree of the user in different fine-grained aspects, and improves the accuracy of recommendation and user experience.

Description

Technical Field

[0001] This invention relates to the field of personalized recommendation technology, and more specifically, to a multi-view sequence recommendation method and system based on multimodal fine-grained information. Background Technology

[0002] With the continuous development of intelligent recommendation systems, how to make accurate product or content recommendations based on users' interests and behaviors has become a hot topic in current research and application. In the past few years, recommendation systems based on collaborative filtering and content recommendation methods have been widely used. These methods usually infer users' interests through historical user behavior data. Although they can provide accurate recommendations to a certain extent, they often ignore the characteristics of the products themselves. In addition, recommendations based on product descriptions often rely on only a single type of information (such as text descriptions) and lack comprehensive consideration of multiple dimensions of products (such as images, categories, brands, etc.). This makes the recommendation performance of the above methods limited when facing complex and diverse recommendation scenarios.

[0003] Multimodal information typically includes features from various types such as text, images, and audio, which help comprehensively describe the attributes of products. However, most existing recommendation methods generally adopt a coarse-grained integration strategy, processing multi-source information only through simple feature concatenation or weighted fusion. This shallow fusion approach fails to fully consider the differences in fine-grained semantic features between different modalities (such as the implicit relationship between material texture in product images and functional attributes in descriptive text), and also tends to ignore hierarchical semantic expressions within the same modality (such as the weight differences between sentiment and entity keywords in user reviews). This results in limited representation capabilities of specific product features, thus affecting recommendation performance. Since knowledge graphs can effectively integrate massive heterogeneous data and domain expert knowledge, embedding them into recommendation models can significantly enhance feature representation capabilities. Therefore, how to integrate knowledge graphs and multimodal fine-grained information to achieve more accurate recommendations is an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a multi-view sequence recommendation method and system based on multimodal fine-grained information. By utilizing a multi-head hidden space for personalized recommendations, it reveals the user's level of interest at different fine-grained levels, thereby improving the accuracy of recommendations and the user experience.

[0005] The first aspect of this invention provides a multi-view sequence recommendation method based on multimodal fine-grained information, comprising the following steps:

[0006] Obtain users' historical product interaction data and product metadata, construct a knowledge graph based on the historical product interaction data and product metadata, and use the knowledge graph to obtain product embedding representations;

[0007] According to the metadata of the commodity, text modal data and image modal data related to the commodity are acquired, and the text modal data and image modal data are respectively fine-grained to generate corresponding fine-grained feature embedding representations;

[0008] The commodity embedding representation and the fine-grained feature embedding representations of different modalities are fused, and a contrast learning strategy is used to optimize the fine-grained fusion to obtain a comprehensive commodity embedding representation;

[0009] The comprehensive commodity embedding representation is mapped to a multi-head hidden space, the user preference is modeled from different fine-grained perspectives, the recommendation probability of the candidate commodity is aggregated for each fine-grained perspective, and a final recommendation probability is obtained, and the next commodity demand of the user is predicted according to the final recommendation probability.

[0010] In the scheme, the historical commodity interaction data and commodity metadata of the user are acquired, and a knowledge graph is constructed based on the historical commodity interaction data and commodity metadata, specifically:

[0011] The historical commodity interaction records of the user and the commodity are extracted from the database and the log, including user ID, commodity ID, interaction time, interaction type, commodity category and brand category information, and detailed description information of the commodity is collected as commodity metadata;

[0012] Based on the historical commodity interaction records and commodity metadata, entities are extracted, user nodes, commodity nodes, category nodes and brand nodes are constructed, and the relationships between entities are defined according to the historical commodity interaction records and commodity metadata, and user-commodity relationships, commodity-category relationships, commodity-brand relationships and commodity-commodity relationships are constructed;

[0013] The extracted entities and relationships are stored in a triple database to form a structured knowledge graph, and the newly added user interaction data and commodity information are regularly updated to the knowledge graph.

[0014] In the scheme, the commodity embedding representation is obtained by using the knowledge graph, specifically:

[0015] The KGAT model is used for node embedding learning, and an initial embedding vector is assigned to each entity in the knowledge graph, and for each target commodity node, the embedding of the one-hop neighbor node is collected, and the attention weight of the neighbor node to the target commodity node is calculated using a self-attention mechanism;

[0016] The embedding of the neighbor node is weighted and summed according to the attention weight to obtain an enhanced embedding representation of the target commodity node, high-order neighbor information is captured by stacking multiple layers of KGAT, and the final embedding representation is obtained for normalization processing to ensure data stability, and the knowledge graph enhanced commodity embedding representation of the target commodity node is output.

[0017] In the scheme, the text modal data and the image modal data are respectively fine-grained to generate corresponding fine-grained feature embedding representations, specifically:

[0018] The text modal data of the commodity is split into fine-grained semantic units Token using a pre-trained BERT model tokenizer, the fine-grained semantic units Token are mapped to unique IDs and converted into word vectors to obtain a corresponding fine-grained semantic unit Token sequence;

[0019] The fine-grained semantic unit Token sequence is input into a BERT model to obtain context-related embedding representations of each fine-grained semantic unit Token through multi-layer Transformer encoding, and output fine-grained embedding vectors of each fine-grained semantic unit Token;

[0020] The frequency of occurrence of the fine-grained semantic unit Token is counted, and high-frequency fine-grained semantic unit Tokens that meet the preset requirements are selected according to the frequency of occurrence, and stop words and low-frequency noise are removed to obtain a simplified text modal fine-grained feature embedding representation;

[0021] The image modal data is segmented into fixed-size image blocks using a pre-trained BEiT model, the encoded image blocks are mapped to discrete visual Tokens using a BEiT pre-trained visual dictionary Codebook, and fine-grained embedding vectors of the visual Tokens are output;

[0022] The frequency of occurrence of the visual Token is counted, and high-frequency visual Tokens that meet the preset requirements are selected according to the frequency of occurrence, and redundant image blocks are filtered to obtain a simplified image modal fine-grained feature embedding representation.

[0023] In the scheme, the commodity embedding representation and the fine-grained feature embedding representation of different modalities are fused, and a contrast learning strategy is used to optimize fine-grained fusion to obtain a comprehensive commodity embedding representation, specifically:

[0024] The knowledge graph enhanced commodity embedding representation, the text modal fine-grained feature embedding representation, and the image modal fine-grained embedding representation are obtained, the embedding representations of different modalities are mapped to the same dimension, and the commodity embedding representation, the text modal fine-grained feature embedding representation, and the image modal fine-grained embedding representation are spliced to form a fusion sequence;

[0025] The fusion sequence is input into a multi-layer Transformer model to realize cross-modal feature interaction through a self-attention mechanism to generate a preliminary fusion representation, and the preliminary fusion representation is subjected to mean pooling operation to obtain a final fusion embedding sequence;

[0026] A contrast learning strategy is introduced, taking the fusion embedding sequence as an anchor point, and the text modality fine-grained feature embedding representation and the image modality fine-grained embedding representation as candidate groups, combining the anchor point and the features in the candidate groups to construct a positive sample pair, and using a batch negative sampling method to construct a negative sample pair;

[0027] The InfoNCE loss function is used to narrow the embedding distance of the positive sample pair and to push away the embedding distance of the negative sample pair, and the comprehensive commodity embedding representation is obtained through contrast learning.

[0028] In the scheme, the comprehensive commodity embedding representation is mapped to a multi-head hidden space, the user preference is modeled from different fine-grained perspectives, the recommendation probability of the candidate commodities is aggregated from each fine-grained perspective, and the final recommendation probability is obtained, specifically:

[0029] The comprehensive commodity embedding representation of the user's historical interaction commodities sorted by time is obtained, a historical interaction series is constructed, the user preference is modeled from different fine-grained perspectives, and M fine-grained perspectives are preset. Each fine-grained perspective maps the comprehensive commodity embedding representation to a dedicated subspace through an independent linear transformation matrix.

[0030] Each fine-grained perspective uses a GRU network to process the user's historical interaction sequence, and the user interest under the m-th fine-grained perspective is summarized through the hidden state, and the user interest under the m-th perspective is represented as is represented as:

[0031] ,

[0032] ,

[0033] wherein, represents the time step updated hidden state, represents the user interacts at the time step comprehensive commodity embedding representation, represents the hidden state of the GRU at the time step encodes the sequence interest of the user up to the time , represents the embedding representation of the second last commodity in the historical interaction sequence, represents the hidden state of the time step ;

[0034] For a candidate commodity , the recommendation probability , at the m-th fine-grained perspective is calculated, the attention weight between perspectives is calculated through an attention mechanism, and the final probability of the fine-grained perspective is obtained through the weighted sum of the probabilities of other perspectives.

[0035] preset view angle weight generate a final recommendation probability , denoted as:

[0036] ,

[0037] wherein denotes the final probability of each fine-grained view angle for the candidate commodity.

[0038] In the scheme, the next commodity demand of the user is predicted according to the final recommendation probability, specifically:

[0039] All candidate commodities are sorted in descending order of final probability to generate an ordered category, and commodities with final probability exceeding a preset probability threshold are reserved to generate a high intention list.

[0040] The next commodity demand of the user is predicted in the high intention list according to the order, and a recommendation is made.

[0041] The second aspect of the present application provides a multi-view sequence recommendation system based on multi-modal fine-grained information, which comprises a user information module, a multi-modal feature module, a fine-grained feature fusion module, a multi-view prediction module and a recommendation decision module.

[0042] The user information module obtains historical commodity interaction data and commodity metadata of the user, and constructs a knowledge graph based on the historical commodity interaction data and commodity metadata.

[0043] The multi-modal feature module obtains commodity embedding representation using the knowledge graph, and fine-grains the text modal data and image modal data of the commodity respectively to generate corresponding fine-grained feature embedding representation.

[0044] The fine-grained feature fusion module fuses the commodity embedding representation and the fine-grained feature embedding representation of different modalities, and optimizes the fine-grained fusion by using a contrast learning strategy to obtain a comprehensive commodity embedding representation fused with different fine-grained information.

[0045] The multi-view prediction module maps the comprehensive commodity embedding representation to a multi-head hidden space, models the user preference from different fine-grained views, aggregates the recommendation probability of each fine-grained view for the candidate commodity, and obtains a final probability.

[0046] The recommendation decision module predicts the next commodity demand of the user according to the final recommendation probability, and outputs the commodities with final recommendation probability exceeding a preset threshold as a list of most willing to buy commodities.

[0047] Compared with the prior art, the present application has the following advantages: ​

[0048] The application constructs a multi-view sequence recommendation method based on a knowledge graph and multi-modal fine-grained information to predict the preference of a user for a next purchase of a product, and realizes accurate prediction of customer preference. When processing multi-modal data, not only the overall features of the modal information are focused on, but also fine-grained semantic unit extraction is focused on. In addition, when fusing multiple modal information, unlike the traditional method of simply splicing or averaging the modal information, contrast learning is adopted to improve the effect of multi-modal information fusion. And the application designs an efficient system for predicting the preference of a user for a next purchase of a product. Based on multi-head hidden space for personalized recommendation, the interest degree of a user in different fine-grained aspects is revealed, and the accuracy of recommendation and user experience are improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments or examples of the present application, the following will briefly introduce the drawings needed to be used in the embodiment or example description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 A flowchart of a multi-view sequence recommendation method based on multi-modal fine-grained information is shown;

[0051] Figure 2 A flowchart of fine-grained processing of text modal data and image modal data is shown;

[0052] Figure 3 A flowchart of personalized recommendation through multi-head hidden space is shown;

[0053] Figure 4 A flowchart of a multi-view sequence recommendation method based on multi-modal fine-grained information is shown;

[0054] Figure 5 A block diagram of a multi-view sequence recommendation system based on multi-modal fine-grained information is shown. DETAILED DESCRIPTION

[0055] In order to more clearly illustrate the technical solutions in the embodiments or examples of the present application, the following will briefly introduce the drawings needed to be used in the embodiment or example description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0056] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0057] As Figure 1 , 4 shown, the embodiment provides a multi-view sequence recommendation method based on multi-modal fine-grained information, comprising:

[0058] S102, obtaining the historical commodity interaction data of the user and the commodity metadata, constructing a knowledge graph based on the historical commodity interaction data and the commodity metadata, and obtaining commodity embedding representation by using the knowledge graph;

[0059] S104, obtaining text modal data and image modal data related to the commodity according to the metadata of the commodity, fine-graining the text modal data and image modal data respectively, and generating corresponding fine-grained feature embedding representation;

[0060] S106, the commodity embedding representation and the fine-grained feature embedding representation of different modalities are fused, and a contrast learning strategy is used to optimize the fine-grained fusion to obtain a comprehensive commodity embedding representation;

[0061] S108, mapping the comprehensive commodity embedding representation to a multi-head hidden space, modeling the user preference from different fine-grained perspectives, aggregating the recommendation probability of the candidate commodity from each fine-grained perspective to obtain a final recommendation probability, and predicting the next commodity demand of the user according to the final recommendation probability.

[0062] It should be noted that the historical commodity interaction records of the user and the commodity are extracted from the database and the log, including user ID, commodity ID, interaction time, interaction type, commodity category and brand category information, and detailed description information of the commodity is collected as commodity metadata, including commodity title, text description, image, price, belonging category, brand, attribute label (such as color, size) and the like. Based on the historical commodity interaction records and the commodity metadata, entities are extracted, and user nodes, commodity nodes, category nodes and brand nodes are constructed, wherein the user node is uniquely identified by the user ID, representing the user in the system, the commodity node is uniquely identified by the commodity ID, containing the basic information of the commodity, the category node represents the classification of the commodity, and the brand node represents the brand of the commodity. The relationship between entities is defined according to the historical commodity interaction records and the commodity metadata, and user-commodity relationship, commodity-category relationship, commodity-brand relationship and commodity-commodity relationship are constructed, wherein the user-commodity relationship represents the interaction behavior between the user and the commodity, the commodity-category relationship represents the category to which the commodity belongs, the commodity-brand relationship represents the brand attribution of the commodity, and the commodity-commodity relationship represents the association between commodities. The extracted entities and relationships are stored in a triple database to form a structured knowledge graph, and the newly added user interaction data and commodity information are updated to the knowledge graph regularly.

[0063] It should be noted that the KGAT model is used for node embedding learning, and an initial embedding vector is assigned to each entity in the knowledge graph. For example, the initial embedding of a commodity node can be generated by the Embedding layer of the commodity ID, and the initial embedding of a user node can be obtained by aggregating the user historical behaviors. The embedding of the one-hop neighbor node is collected for each target commodity node, and the attention weight of the neighbor node to the target commodity node is calculated using the self-attention mechanism. The attention weight reflects the influence degree of the neighbor to the target node. The embedding of the neighbor node is weighted and summed according to the attention weight, and the enhanced embedding representation of the target commodity node is obtained, which is represented as:

[0064] ,

[0065] wherein, represents the initial embedding of the node , represents the embedding of the neighbor node , represents the edge set, represents the self-attention mechanism, is the enhanced embedding representation of the commodity node ;

[0066] By stacking multiple layers of KGAT, high-order neighbor information can be captured. For example, the node embedding of the second layer will integrate the representation of its neighbor aggregated in the first layer, thereby enhancing the transmission of global information. Multi-layer KGAT can capture the indirect association between commodities. The final embedding representation is obtained and normalized to ensure data stability, and the knowledge graph enhanced commodity embedding representation of the target commodity node is output. The KGAT model dynamically allocates weights to entities and their relationships in the graph through the self-attention mechanism, thereby capturing the collaborative information and characteristic association between commodities, forming a more rich and context-related commodity embedding representation, which provides a strong semantic foundation for subsequent multi-modal fusion and sequence recommendation.

[0067] Figure 2 A flowchart for fine-grained processing of text modal data and image modal data is shown.

[0068] According to an embodiment of the present application, the text modal data and image modal data are fine-grained respectively, and corresponding fine-grained feature embedding representations are generated. Specifically,

[0069] S202, the text modal data of the commodity is split into fine-grained semantic units Token using a pre-trained BERT model tokenizer, the fine-grained semantic units Token are mapped to unique IDs, and converted into word vectors to obtain a corresponding fine-grained semantic unit Token sequence;

[0070] S204, input the fine-grained semantic unit Token sequence into the BERT model, obtain the context-related embedding representation of each fine-grained semantic unit Token through multi-layer Transformer encoding, and output the fine-grained embedding vector of each fine-grained semantic unit Token;

[0071] S206, count the frequency of occurrence of the fine-grained semantic unit Token, filter high-frequency fine-grained semantic unit Tokens that meet the preset requirements according to the frequency of occurrence, remove stop words and low-frequency noise, and obtain a simplified text modal fine-grained feature embedding representation;

[0072] S208, using the pre-trained BEiT model, the image modal data is segmented into fixed-size image blocks, the image blocks are encoded, and the encoded image blocks are mapped into discrete visual Tokens using the BEiT pre-trained visual dictionary Codebook, and the fine-grained embedding vector of the visual Token is outputted;

[0073] S210, count the frequency of occurrence of the visual Token, filter high-frequency visual Tokens that meet the preset requirements, filter redundant image blocks, and obtain a simplified image modal fine-grained feature embedding representation.

[0074] It should be noted that the use of the text modal data of the product, the text description of the product is segmented using the tokenizer of the BERT model. The tokenizer splits the text modal data into a Token sequence suitable for processing by the BERT model, maps the text description to a set of discrete token ID sequences, and is represented as:

[0075] ,

[0076] wherein, the i-th Token represents the i-th Token of the product , and denotes the number of Tokens of the text modal data.

[0077] For the image modal data of the product, the BEiT model is used for processing. The BEiT model is a pre-trained model based on ViT. The image modal data of the product is segmented into fixed-size blocks , and the image blocks are encoded. Each image block is converted into a Patch Embedding through linear projection, and the Token sequence representation of the image is obtained. The Token sequence representation of the image represents the fine-grained information of the image, and the Token sequence representation is as follows: ​​

[0078] ,

[0079] ,

[0080] in, Indicates goods The Each token represents, The number of tokens representing the image modality data. Indicates the first Image blocks.

[0081] During the tokenization process, duplicate tokens are encountered because certain subwords may appear multiple times in a sentence, and similar semantic elements may appear repeatedly in entity images. Therefore, it is necessary to calculate the frequency of each token and retain a predetermined number of the most common tokens for each modality. Furthermore, stop words with minimal co-occurrence with entity semantics in the text description need to be removed, and the top 10 most frequent text tokens for each product need to be retained. and Image tags The multimodal data of the products is transformed into a high-information, fine-grained embedded representation, providing accurate input for subsequent cross-modal fusion and user interest modeling.

[0082] It should be noted that by obtaining knowledge graph-enhanced product embedding representations, text modality fine-grained feature embedding representations, and image modality fine-grained embedding representations, the embeddings of different modalities are aligned to the same dimension and then concatenated to form the product embedding representation. Fine-grained feature embedding representation of text modalities and fine-grained embedding representation of image modalities A fused sequence is formed; the fused sequence is input into a multi-layer Transformer model, and cross-modal feature interaction is achieved through a self-attention mechanism. Each position vector output by the Transformer contains global context information, generating a preliminary fused representation. The initial fusion representation is then subjected to mean pooling to obtain the final fusion embedding sequence.

[0083] Introducing a contrastive learning strategy to fuse the embedded sequences Using text modality fine-grained feature embedding representation and image modality fine-grained feature embedding representation as anchors, they serve as candidate groups. The anchor points are combined with features from the candidate group to construct positive sample pairs. Consider it as a positive sample pair, where , for example, different modal feature groups of the same commodity, the fusion embedding of the commodity and the text modal fine-grained feature embedding of the commodity constitute a positive pair, and a batch negative sampling method is used to construct a negative sample pair, for example, the embedding of different commodities in the same batch is randomly combined, and the fusion embedding of the commodity and the text modal fine-grained feature embedding of the commodity represent a negative pair. Taking InfoNCE as the contrast backbone, the InfoNCE loss function is used to reduce the embedding distance of the positive sample pair and to increase the embedding distance of the negative sample pair, thereby improving the representation ability of the model to multi-modal fine-grained features. Through contrast learning, a comprehensive commodity embedding representation is obtained, and the InfoNCE loss function is represented as:

[0084] ,

[0085] wherein, represents the batch size, and represents the number of commodities in each batch of data, represents the cosine similarity of two embeddings, is a temperature hyperparameter for adjusting the sensitivity of the contrast loss, represents a positive sample pair, represents a negative sample pair.

[0086] Figure 3 A flowchart for personalized recommendation through multi-head hidden space is shown.

[0087] According to the embodiment of the present application, the comprehensive commodity embedding representation is mapped to the multi-head hidden space, the user preference is modeled from different fine-grained perspectives, the recommendation probability of the candidate commodity is aggregated from each fine-grained perspective, and the final recommendation probability is obtained. Specifically,

[0088] S302, obtain the comprehensive commodity embedding representation of the user's historical interaction commodities sorted by time, construct a historical interaction series, model the user preference from different fine-grained perspectives, and preset M fine-grained perspectives. Each fine-grained perspective maps the comprehensive commodity embedding representation to a dedicated subspace through an independent linear transformation matrix.

[0089] In the prediction phase, multiple heads are used to learn the user's interest from different fine-grained perspectives. First, according to the user's historical interaction data, including user ID, item ID and interaction time, the user's historical interaction commodities are sorted according to the interaction time, and the user's historical interaction sequence is obtained. The is input into each head to learn the representation of the sequence using a GRU model and to predict the recommendation probability of the next commodity. Preset M fine-grained perspectives, for example, M = 4, corresponding to performance, appearance, price and brand.

[0090] S304, for each fine-grained view, a GRU network is used to process the user's historical interaction sequence, summarizing the user's interests under the fine-grained view m through the hidden states; for each head m, a GRU network is used to process the user's historical interaction sequence. , Indicates goods The comprehensive product embedding representation, for the m-th viewpoint, time step Hidden state The update method is as follows:

[0091] ,

[0092] User interest representation is obtained from the final sequence representation of the m-th perspective. Represented as:

[0093] ,

[0094] in, Indicates time step The updated hidden status integrates current product information and historical sequence memory. Indicates user At time step Interactive integrated product embedding representation, This indicates that GRU is at time step The hidden state, encoded by the user up to Sequence interest at time, This represents the embedding representation of the second-to-last item in the historical interaction sequence. Indicates the first The hidden state at each time step.

[0095] S306, regarding candidate products Calculate the recommendation probability at the m-th fine-grained viewpoint. , The attention weights between viewpoints are calculated through an attention mechanism, and the final probability of a fine-grained viewpoint is obtained by weighting the probabilities of other viewpoints.

[0096] Since the perspectives of each viewpoint are not independent, they are interdependent and interactive. An attention mechanism is introduced to calculate the interdependence weights between viewpoints, weighting the probabilities generated by different views to obtain the recommendation probability of the current viewpoint for the product after incorporating other viewpoints. The final probability of each viewpoint k is the weighted sum of the probabilities of the other viewpoints, expressed as:

[0097] ,

[0098] in, The attention weight of the kth view to the mth view is represented, and the calculation formula is as follows:

[0099] ,

[0100] Wherein The fine-grained view sequence number is represented, The final sequence representation of the kth view is represented, The transpose of the final sequence representation of the mth view is represented, The transpose of the final sequence representation of the mth view is represented.

[0101] S308, preset view weight Combine the final probability of each fine-grained view to generate the final recommendation probability , represented as:

[0102] ,

[0103] Wherein The final probability of each fine-grained view to the candidate commodity .

[0104] It should be noted that all candidate commodities are sorted according to the final probability from high to low to generate an ordered category, and the commodities with a final probability exceeding a preset probability threshold are retained to generate a high intention list; in the high intention list, the next commodity demand of the user is predicted according to the order and recommended. According to business requirements, dynamically adjust the threshold, reduce the number of recommendations and improve accuracy in strict mode by lowering the threshold; in the relaxed mode, lower the threshold to increase the diversity of recommendations; for new users or low active users, temporarily lower the threshold to expand the candidate range, and subsequently calibrate according to feedback, limit the number of commodities of the same category or brand (such as a maximum of 2 recommended for each brand) in the high intention list to avoid excessive concentration. Retain some commodities with slightly lower probability than the threshold but with novelty (such as new products and categories that the user has not contacted) to enrich the selection.

[0105] Figure 5 A block diagram of a multi-view sequence recommendation system based on multi-modal fine-grained information is shown.

[0106] The second embodiment of the application provides a multi-view sequence recommendation system 5 based on multi-modal fine-grained information, comprising: a user information module 501, a multi-modal feature module 502, a fine-grained feature fusion module 503, a multi-view prediction module 504 and a recommendation decision module 505.

[0107] ​The user information module obtains historical commodity interaction data and commodity metadata of a user, constructs a knowledge graph based on the historical commodity interaction data and the commodity metadata, constructs the knowledge graph based on a historical purchase sequence of the user, captures collaborative information and characteristic correlations between commodities, and forms a more rich and context-related commodity embedding representation.

[0108] The multi-modal feature module obtains commodity embedding representations using the knowledge graph, respectively performs fine-grainedization on text modal data and image modal data of the commodities, generates corresponding fine-grained feature embedding representations, obtains fine-grained information under the multi-modal of the commodities, and fuses features of different modalities to obtain a commodity embedding representation fused with multiple fine-grained features.

[0109] The fine-grained feature fusion module fuses the commodity embedding representation and the fine-grained feature embedding representations of different modalities, optimizes fine-grained fusion by using a contrast learning strategy, obtains a comprehensive commodity embedding representation fused with different fine-grained information, and fuses the commodity embedding enhanced by the knowledge graph and information of different modalities to obtain a comprehensive embedding representation of the commodity fused with different fine-grained information.

[0110] The multi-view prediction module maps the comprehensive commodity embedding representation to a multi-head hidden space, models user preferences from different fine-grained perspectives, aggregates recommendation probabilities of candidate commodities from each fine-grained perspective to obtain a final probability, maps comprehensive commodity embeddings in historical interaction data of a user to multiple multi-head hidden spaces, explores commodity sequences from different perspectives, more accurately predicts next commodity demands of the user, and thus optimizes recommendation results.

[0111] The recommendation decision module predicts next commodity demands of the user according to the final recommendation probability, and outputs commodities with a final recommendation probability exceeding a preset threshold as a list of commodities most willing to be purchased.

[0112] The third aspect of the present application provides a computer readable storage medium, the computer readable storage medium comprises a multi-view sequence recommendation method based on multi-modal fine-grained information program, when the multi-view sequence recommendation method based on multi-modal fine-grained information program is executed by a processor, the steps of the multi-view sequence recommendation method based on multi-modal fine-grained information are realized.

[0113] In several embodiments provided in the present application, it should be understood that the disclosed method and system can be implemented in other ways. The above-described system embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms. In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0114] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes mobile storage equipment, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various storage program codes.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-view sequence recommendation method based on multimodal fine-grained information, characterized in that, Includes the following steps: Obtain users' historical product interaction data and product metadata, construct a knowledge graph based on the historical product interaction data and product metadata, and use the knowledge graph to obtain product embedding representations; Based on the product's metadata, obtain the product-related text modal data and image modal data, and refine the text modal data and image modal data respectively to generate corresponding fine-grained feature embedding representations; The product embedding representation and the fine-grained feature embedding representations of different modalities are fused together, and a contrastive learning strategy is used to optimize the fine-grained fusion to obtain a comprehensive product embedding representation. The comprehensive product embedding representation is mapped to a multi-head hidden space, user preferences are modeled from different fine-grained perspectives, the recommendation probabilities of candidate products from each fine-grained perspective are aggregated to obtain the final recommendation probability, and the user's next product demand is predicted based on the final recommendation probability. The text modal data and image modal data are respectively refined to generate corresponding fine-grained feature embedding representations, specifically as follows: The text modal data of the product is split into fine-grained semantic units (Tokens) using a pre-trained BERT model word segmenter. The fine-grained semantic unit Tokens are mapped to unique IDs and converted into word vectors to obtain the corresponding fine-grained semantic unit Token sequence. The sequence of fine-grained semantic units (Tokens) is input into the BERT model. The context-related embedding representation of each fine-grained semantic unit (Token) is obtained through multi-layer Transformer encoding, and the fine-grained embedding vector of each fine-grained semantic unit (Token) is output. The frequency of occurrence of fine-grained semantic unit tokens is statistically analyzed. High-frequency fine-grained semantic unit tokens that meet the preset requirements are selected based on the frequency of occurrence. Stop words and low-frequency noise are removed to obtain a concise text modality fine-grained feature embedding representation. The image modal data is segmented into fixed-size image blocks using a pre-trained BEiT model. The image blocks are then encoded, and the encoded image blocks are mapped to discrete visual tokens using the BEiT pre-trained visual dictionary Codebook. Fine-grained embedding vectors of the visual tokens are then output. The frequency of occurrence of visual tokens is statistically analyzed, and high-frequency visual tokens that meet preset requirements are selected based on the frequency of occurrence. Redundant image blocks are filtered out to obtain a simplified image modality fine-grained feature embedding representation. The product embedding representation and fine-grained feature embedding representations of different modalities are fused, and a contrastive learning strategy is used to optimize the fine-grained fusion to obtain a comprehensive product embedding representation, specifically: We acquire knowledge graph-enhanced product embedding representations, text modality fine-grained feature embedding representations, and image modality fine-grained embedding representations. We then unify the embeddings of different modalities to the same dimension and concatenate the product embedding representations, text modality fine-grained feature embedding representations, and image modality fine-grained embedding representations to form a fusion sequence. The fused sequence is input into a multi-layer Transformer model, and cross-modal feature interaction is achieved through a self-attention mechanism to generate a preliminary fused representation. The preliminary fused representation is then subjected to mean pooling to obtain the final fused embedding sequence. A contrastive learning strategy is introduced, using the fused embedding sequence as the anchor point, and the text modality fine-grained feature embedding representation and the image modality fine-grained feature embedding representation as candidate groups. The anchor point and the features in the candidate groups are combined to construct positive sample pairs, and a batch negative sampling method is used to construct negative sample pairs. The InfoNCE loss function is used to narrow the embedding distance of positive sample pairs and widen the embedding distance of negative sample pairs. A comprehensive product embedding representation is obtained through contrastive learning.

2. The multi-view sequence recommendation method based on multimodal fine-grained information according to claim 1, characterized in that, Obtain users' historical product interaction data and product metadata, and construct a knowledge graph based on the historical product interaction data and product metadata, specifically as follows: Extract historical product interaction records between users and products from the database and logs, including user ID, product ID, interaction time, interaction type, product category and brand category information, and collect detailed description information of products as product metadata; Based on the historical product interaction records and extracted product metadata entities, user nodes, product nodes, category nodes, and brand nodes are constructed. Furthermore, based on the relationships between entities defined by the historical product interaction records and product metadata, user-product relationships, product-category relationships, product-brand relationships, and product-product relationships are constructed. The extracted entities and relationships are stored in a triple database to form a structured knowledge graph, and new user interaction data and product information are regularly updated into the knowledge graph.

3. The multi-view sequence recommendation method based on multimodal fine-grained information according to claim 1, characterized in that, The product embedding representation is obtained using the knowledge graph, specifically as follows: The KGAT model is used for node embedding learning. An initial embedding vector is assigned to each entity in the knowledge graph. For each target product node, the embeddings of one-hop neighbor nodes are collected. The self-attention mechanism is used to calculate the attention weights of neighbor nodes to the target product node. The embeddings of neighboring nodes are weighted and summed according to the attention weights to obtain the enhanced embedding representation of the target product node. High-order neighbor information is captured by stacking multiple layers of KGAT. The final embedding representation is then normalized to ensure data stability and output the knowledge graph-enhanced product embedding representation of the target product node.

4. The multi-view sequence recommendation method based on multimodal fine-grained information according to claim 1, characterized in that, The comprehensive product embedding representation is mapped to a multi-headed latent space, and user preferences are modeled from different fine-grained perspectives. The recommendation probabilities of candidate products from each fine-grained perspective are aggregated to obtain the final recommendation probability, specifically: Obtain the comprehensive product embedding representation of the historical interaction products of the user sorted by time, construct the historical interaction series, model user preferences from different fine-grained perspectives, preset M fine-grained perspectives, and map the comprehensive product embedding representation to a dedicated subspace through an independent linear transformation matrix for each fine-grained perspective; Each fine-grained view uses a GRU network to process the user's historical interaction sequence, summarizing the user interests under the fine-grained view m through the hidden states, and representing the user interest of the m-th view. Represented as: ， in, This represents the embedding representation of the second-to-last item in the historical interaction sequence. Indicates the first The hidden state at time step n, encoded by the user up to the nth time step n. Sequence interest at each time step; For candidate products Calculate the recommendation probability at the m-th fine-grained viewpoint. , The attention weights between viewpoints are calculated through an attention mechanism, and the final probability of a fine-grained viewpoint is obtained by weighting the probabilities of other viewpoints. Preset view weights The final recommendation probability is generated by combining the final probabilities from each fine-grained perspective. , is represented as: ， in This indicates that each fine-grained perspective is relevant to the candidate product. The final probability.

5. The multi-view sequence recommendation method based on multimodal fine-grained information according to claim 1, characterized in that, The user's next product demand is predicted based on the final recommendation probability, specifically as follows: Sort all candidate products from high to low according to their final probability, generate ordered categories, and retain products whose final probability exceeds a preset probability threshold to generate a high-intent list; Based on the ranking of the high-intent list, predict the user's next product demand and make recommendations.

6. A multi-view sequence recommendation system based on multimodal fine-grained information, characterized in that, The system implements the multi-view sequence recommendation method based on multimodal fine-grained information as described in any one of claims 1-5, comprising a user information module, a multimodal feature module, a fine-grained feature fusion module, a multi-view prediction module, and a recommendation decision module; The user information module acquires the user's historical product interaction data and product metadata, and constructs a knowledge graph based on the historical product interaction data and product metadata; The multimodal feature module uses knowledge graphs to obtain product embedding representations, and performs fine-grained processing on the text modal data and image modal data of the products to generate corresponding fine-grained feature embedding representations. The fine-grained feature fusion module fuses the product embedding representation and the fine-grained feature embedding representations of different modalities, and uses a contrastive learning strategy to optimize the fine-grained fusion to obtain a comprehensive product embedding representation that fuses different fine-grained information. The multi-view prediction module maps the comprehensive product embedding representation to a multi-head hidden space, models user preferences from different fine-grained perspectives, aggregates the recommendation probability of each fine-grained perspective for candidate products, and obtains the final probability. The recommendation decision module predicts the user's next product demand based on the final recommendation probability, and outputs the products whose final recommendation probability exceeds a preset threshold as the list of products most likely to be purchased.

Images