Wine recommendation method and system based on multi-modal heterogeneous data

Abstract

The invention discloses a wine recommendation method and system based on multi-modal heterogeneous data. The method comprises the following steps: constructing wine attributes and customer feature missing information based on fine adjustment LLM for wine and customers thereof; according to the information generated by the fine-tuning LLM, using a pre-trained Transform model to convert customer features and wine information from original forms into digital embedded vectors to replace random embedded vectors; constructing a customer-grape wine social hypergraph, and quantifying the weight of the grape wine and the customer in the social relationship by integrating the global and local importance of the grape wine and the customer in the hypergraph; and according to the hypergraph, extracting a sub-graph with an actual interaction behavior to construct a user-wine isograph to enhance learning of customer positive behaviors, and the like. According to the embodiment of the invention, the influence of the customer characteristics and the implicit social relationship under the interactive behavior on the grape wine purchasing behavior is deeply mined, and the grape wine recommendation precision based on the multi-modal heterogeneous data is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of consumer product recommendation, and in particular to a wine recommendation method and system based on multi-modal heterogeneous data. BACKGROUND

[0002] The vast number of wine products makes it time-consuming and laborious for customers to choose the wine they like. Therefore, it is very important to accurately identify the personalized preferences of customers and recommend suitable wines to them.

[0003] Traditional recommendation algorithms are based on single-modal interaction data, which lack understanding of the complex characteristics of wine and have limitations in practical applications. The development of the digital era provides convenience for collecting multi-modal wine consumption data. However, the multi-modal attributes of wine make the recommendation algorithm need to deal with a large amount of unstructured data; and because many customers have only purchased a few types of wine, the interaction data between customers and wine often presents sparsity. This unstructured and sparse nature makes it difficult for recommendation algorithms to accurately capture customers' interests and preferences. Therefore, effectively mining sparse rating data and rich unstructured data has important theoretical and practical significance for realizing personalized wine recommendation based on multi-modal heterogeneous data.

[0004] Existing recommendation algorithms for unstructured and sparse data have made significant developments in terms of collaborative filtering, graph and knowledge graph, and data augmentation. The emergence of multi-modal recommendation, large language models and other technologies also provides a cutting-edge perspective for dealing with the sparsity of unstructured data, but most of them are applied to mixed product recommendations, and the research on wine recommendation is still in its early stages.

[0005] With the development of graph neural networks, some scholars have used interactive data and multimodal data to accurately capture customer preferences for wine through feature extraction and higher-order fusion. They directly extract features from each modality (e.g., using CNN to extract image features, or BERT to extract text features) and merge them into a unified feature vector, which is then input into the graph neural network for training. Other scholars have integrated side information (such as customer attributes, item features, and social networks) into the graph structure to improve recommendation quality using existing structured information and relationships. During model training, side information is used to adjust the latent representations of customers and wines for model reinforcement learning. While existing algorithms such as DualGNN and DRAGON utilize graphs to better handle the sparsity and unstructured characteristics of multimodal data, these graphs mine social relationships by calculating the similarity of user interactions, neglecting the influence of indirect social relationships between consumers and their direct and indirect friends, as well as the influence of implicit social relationships between purchased items. Furthermore, these algorithms only use user nodes, item nodes, and user-item interaction relationships to construct the graph, without introducing user features and item attributes as attribute nodes into the graph. In fact, these attribute nodes play an enhancing role in the graph. Therefore, there is an urgent need for a wine recommendation method based on multimodal heterogeneous data to address the aforementioned issues. Summary of the Invention

[0006] One technical problem addressed by this invention is the lack of accuracy in existing wine recommendations. It provides a wine recommendation method and system based on multimodal heterogeneous data, which can solve the difficulties in existing multimodal recommendation technologies that ignore the influence of customer characteristics on purchasing behavior and the implicit social meaning under interaction behavior, thereby improving the accuracy of wine recommendations based on multimodal heterogeneous data.

[0007] The technical solution adopted in this invention is: a wine recommendation method based on multimodal heterogeneous data, comprising the following steps: Step S100, constructing wine attribute and customer feature missing information based on fine-tuned LLM for wine and its customers; Step S200, based on the information generated by fine-tuned LLM in S100, using a pre-trained Transformer model to convert customer features and wine information from their original form into digital embedding vectors instead of random embedding vectors; the customer features and wine information include images and text; Step S300, constructing a customer-wine social hypergraph, integrating wine... Step S400: The weight of a customer in social relationships is quantified by the global and local importance of the customer in the hypergraph; Step S500: A user-wine isomorphic graph is constructed by extracting subgraphs with actual interaction behaviors from the social hypergraph to enhance the learning of positive customer behaviors; Step S600: The captured social weights and relationships are injected into the embedding vectors of customers and wines, and a multimodal information fusion representation of customer preferences is achieved; Step S600: In the prediction stage, positive feedback, negative feedback, and neutral feedback states are proposed based on the subgraphs with actual interaction behaviors to improve the BPR loss function, and the ranking results of the wines that customers may like are recommended.

[0008] A wine recommendation system based on multimodal heterogeneous data includes:

[0009] A missing information generation module, which generates missing wine attributes and customer characteristics based on a fine-tuned LLM;

[0010] The interaction perception preference mining module, based on the customer-wine social hypergraph and the user-wine isomorphic graph, injects the captured social weights and relationships into customer and wine nodes, and realizes the multimodal information fusion representation of customer preferences;

[0011] The prediction module optimizes the loss function based on the customer's interaction with the wines to obtain the final ranked wine recommendation results.

[0012] As a further improvement of the present invention, step S100 includes: retrieving an existing publicly available wine dataset 1; constructing a wine attribute generation Prompt based on the wine attribute information in the aforementioned dataset 1, and using the Prompt to fine-tune the large-scale language model LLM to generate missing wine attributes in the dataset; based on the rating data and complete wine attribute data of the aforementioned dataset 1, statistically analyzing the types of wine attributes preferred by each customer; retrieving a different dataset 2; calculating the similarity of wine attribute preferences between customers in dataset 2 and customers in dataset 1 based on cosine similarity, and selecting the top k customers; constructing a customer personal feature generation Prompt, and using the Prompt to fine-tune the large-scale language model LLM; the fine-tuned LLM, as a generative model, can effectively generate the missing customer personal features in dataset 1, including gender, age, marital status, occupation, monthly disposable income, and education level, thereby enhancing customer information.

[0013] As a further improvement of the present invention, step S200 includes: using a pre-trained Transformer MiniLM model to convert the original information of the text modal data of customer feature information and the text modal data of wine attributes that were supplemented in S100 into numerical embedding vectors; using a CLIP model to convert the original information of the image modal data of wine attributes into numerical embedding vector representations; and in the input stage, replacing the customer random feature embedding with the embedding vector of customer feature information as the initial representation of the customer, and replacing the item ID embedding vector in the wine attributes with the feature embedding vector of wine as the initial representation of the wine.

[0014] As a further improvement of the present invention, step S300 includes: defining three types of edges and six types of hyperedges based on customer characteristics, wine attributes and customer-wine interaction behavior, and constructing a customer-wine social hypergraph; calculating the global importance and local importance of each customer and wine based on the social edges in the customer-wine social hypergraph; and combining the global importance and local importance to obtain the social weight of the customer and the item.

[0015] As a further improvement of the present invention, the global importance is calculated using the PageRank algorithm, and the local importance is calculated using degree centrality.

[0016] As a further improvement of the present invention, the step of combining global importance and local importance to obtain the social importance of customers and items includes: arranging the global importance and local importance of each customer and wine to generate a matrix list, which serves as the social weight of customers and items.

[0017] As a further improvement of the present invention, step 400 includes: extracting customer-item interaction behavior scores from the customer-wine social hypergraph to construct an isomorphic graph based on customer-item interaction behavior, and setting an isomorphic graph threshold; if the number of wines interacted by the customer is greater than the threshold, then selecting the wines with the highest scores within the threshold quantity; if the number of wines interacted by the customer is less than the isomorphic graph threshold, then randomly selecting from the interacted items to make the quantity reach the threshold, thereby achieving the purpose of isomorphism.

[0018] As a further improvement of the present invention, step S500 includes: fusing multimodal information of customers and wine respectively; using the Attentive Concatenation method for the multimodal embedding of wine, the features of each modality are weighted and fused according to the user's preference for different modalities, with the weights initialized to 0.5; the multimodal embedding of customers is directly concatenated, simply concatenating the information of each modality together; and forming their integrated representation based on the representations of customers and wine learned from the customer-wine social hypergraph and the user-wine isomorphic graph, as well as the importance of customers and items calculated from the customer-wine social hypergraph.

[0019] As a further improvement of the present invention, step S600 includes: setting the relationship between the customer and the wine that the customer has not rated as a negative state according to the interaction state; setting a threshold for wine rating; defining the interaction behavior above the threshold as a positive state and the interaction behavior below the threshold as a neutral state; generating sample pairs respectively; and maximizing the difference in user preference ranking between positive samples and negative pairs and between neutral samples and negative samples to obtain the optimized loss function.

[0020] The beneficial effects of this invention are: the recommendation method of this invention solves the difficulties of ignoring the impact of customer characteristics on purchasing behavior and ignoring the implicit social meaning under interaction behavior, thereby improving the accuracy of wine recommendation based on multimodal heterogeneous data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a wine recommendation method based on multimodal heterogeneous data provided by the present invention;

[0023] Figure 2This invention provides a schematic diagram of the structure of a wine recommendation system based on multimodal heterogeneous data. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The wine recommendation method based on multimodal heterogeneous data provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0026] In this example, Dataset 1 uses the X-Wines dataset, which includes rating data, wine textual attribute data, and wine image data. Dataset 2 uses the grape_wines dataset, which comes from data collected by our research group from June to October 2020 for the National Grape Industry Technology System survey. It has 3621 valid samples. Specific data information is shown in Table 4 and its description below. In this example, the interactive behavior refers to rating behavior; the item refers to wine; and the consumer refers to the customer.

[0027] The X-Wines dataset was collected by de Azambuja et al. from multiple professional wine websites on open networks between February and March 2022. Its data source is published in the official repository of the X-Wines project (https: / / github.com / rogerioxavier / X-Wines). This dataset contains both textual and visual features (product descriptions and images) as well as rating data, and is divided into three different versions. Their statistical information is shown in Table 1. For computational efficiency, the X-Wines slim version was used in the verification phase of this invention. The wine attributes and their descriptions in the X-Wines dataset used in this invention are shown in Table 2, and the variables and their descriptions in the rating data used are shown in Table 3.

[0028] Table 1. Statistics on different versions of the X-Wines dataset

[0029]

[0030] Table 2. Wine Attributes and Descriptions

[0031] Wine attributes Description Wine ID Integer. The primary key identifying the wine. Wine name String. The textual representation of the wine label as presented on the label. Type String. The classification type definition: still wines are divided into red, white or rosé; sweet and fortified wines are divided into sparkling or dessert wines. Dessert / port wine is a subcategory of fortified wines. Production information String. The classification category used to distinguish single variety or blended / blend wines. The most famous blended wines are also taken into account, such as Bordeaux red and white blends, Valpolicella blends and Portuguese red and white blends. Grape varieties used String list. Contains the grape varieties used for the wine production, preserving the original names. Food pairing String list. Contains the main courses combinations that go well with the wine, these pairings are provided by the producer, as well as publicly recommended by sommeliers or even consumers on the internet. Alcohol content (ABV) Float. Represents the alcohol by volume (ABV). The value shown on the label can be inaccurate, allowing a margin of error of 0.5% per 100 volume units, with some wines having a tolerance of up to 0.8%. Body String. Classification of the wine body based on its viscosity: very light, light, medium, full or very full. Acidity String. Classification of the wine acidity based on the potential hydrogen (pH) value: low, medium or high Country String. Classification of the country of origin of the wine (ISO-3166 standard). Region name String. Textual identification of the wine's region of origin. When the name of the designated region is identified, it is preserved. Estate name String. Textual information identifying the winery. Year String list. Contains a list of integers representing the year, or the abbreviation "N.V." (meaning "non-vintage")

[0032] Table 3. Description of scoring data

[0033]

[0034] The grape_wines dataset used in this invention comes from the survey data of the National Grape Industry Technology System, with a total of 3621 valid samples collected from June to October 2020. The data content is divided into two parts: customer preferences for wine sensory attributes and demographic characteristics. The wine sensory attributes include color, effervescence, aroma, acidity, sweetness, and taste, while the demographic characteristics include gender, age, marital status, occupation, monthly disposable income, and education level, as detailed in Table 4.

[0035] Table 4. Description of attributes in the grape_wines dataset

[0036]

[0037] A wine recommendation method based on multimodal heterogeneous data includes:

[0038] Step 100: Construct missing wine attributes and customer characteristics information based on a fine-tuned LLM for wine and its customers.

[0039] While graph-structured multimodal recommender systems offer significant advantages in managing unstructured and sparse data, they typically focus on the multimodal attributes of goods and the interaction between consumers and wines, neglecting the impact of customer characteristics on purchasing behavior. Therefore, embodiments of this invention optimize the input information for recommendations by generating missing customer characteristics and wine attributes, including:

[0040] Domain knowledge is infused into the collective information contained in the basic LLM (Local Language Model) to improve the quality of suggestions. This LLM provides information about the ongoing conversation in the form of prompts. To this end, the OpenAI ERNIE-3.5-8K model was fine-tuned on training samples consisting of prompts (input) and completion (expected output). The parameter Temperature=0.8 was set to control the diversity of generated text, and Top_p=0.8 was set to further ensure the quality and relevance of the generated text by limiting the vocabulary selection range. Missing information generation based on the fine-tuned LLM involves two steps: first, generating missing wine attributes based on the fine-tuned LLM; after completing the wine attributes, further generating customer-specific characteristics based on the similarity of wine attributes according to customer preferences using the fine-tuned LLM.

[0041] Based on wine attribute information from the X-Wine dataset, a wine attribute generation prompt was constructed. The wine attribute generation prompt statement for a specific wine is shown below:

[0042] {“Prompt”:

[0043] "The properties and attribute values ​​of this wine are as follows:"

[0044] Wine Name: Haut-Médoc

[0045] Type: Red

[0046] Production Information: Blend / Bordeaux Red blend

[0047] The grapes used for winemaking are: ['Merlot', 'Cabernet Sauvignon'].

[0048] Recommended beverage pairings: [Beef, Lamb, Game, Poultry]

[0049] Alcohol content: 12.5%,

[0050] Body: Full-bodied.

[0051] Acidity: High

[0052] Country: France

[0053] Region name: Haut-Médoc

[0054] Winery Name: Château Coughland

[0055] Years: [2020, 2019, 2018, 2017, 2016, 2015, 2014].

[0056] Please infer the color, effervescence, sweetness, and aroma of the wine. Output the following information about the wine in the following format: {Taste: taste, Color: color, Effervescence: effervescent or still, Sweetness: sweet or dry or semi-dry or semi-sweet, Aroma: floral or fruity or oaky or vegetal or other}. Do not output the reasoning process.

[0057] Based on this prompt, the Large Scale Language Model (LLM) is fine-tuned. The fine-tuned LLM, as a generative model, can effectively complete the missing wine attributes in the dataset, thereby enhancing the wine attribute information, as shown in Equation 1. This indicates the generated wine text attributes. Hints for generating missing information for wine i.

[0058] (1)

[0059] The generation of customer features relies on a fundamental assumption: customers with similar personal characteristics also exhibit a high degree of similarity in their wine attribute preferences. Based on this assumption, this embodiment of the invention first completes the missing wine attributes, then identifies similar customers based on the similarity of their wine attribute preferences. Subsequently, by combining the personal characteristics of similar customers, the missing customer personal features are generated. This process mainly includes the following three steps: First, based on X-Wines rating data and complete wine attribute data, the types of wine attributes preferred by each customer are statistically analyzed; second, based on cosine similarity, the similarity of wine attribute preferences between customers in the grape_wines dataset and customers in the X-Wine dataset is calculated, and the top k customers are selected; then, a customer personal feature generation Prompt is constructed, as shown below:

[0060] {“Prompt”:

[0061] The characteristics of customer 1-k are as follows: {

[0062] Client 1 characteristics: Male, 38 years old, No, Freelancer, 5000-8000, College degree or above; ......;

[0064] Customer K characteristics: Female, 25 years old, no, student, 3000-5000, college degree or above;

[0065] }

[0066] Please generate information for consumer k+1 based on the information above.

[0067] Based on this prompt, the Large Scale Language Model (LLM) is fine-tuned. Finally, the fine-tuned LLM, as a generative model, can effectively generate missing customer personal characteristics in Dataset 1, including gender, age, marital status, occupation, monthly disposable income, and education level, thereby enhancing customer information, as shown in Equation 2. This represents the generated customer-specific text features. This is used to generate prompts for missing information for customer j.

[0068] (2)

[0069] Step 200: Based on the information generated by the fine-tuned LLM, use a pre-trained Transformer model to convert customer features and wine information (including images and text) from their original form into digital embedding vectors instead of random embedding vectors.

[0070] In embodiments of the present invention, the initial embedding vectors for customers and wine are improved. This addresses the shortcoming of using random customer embedding vectors, which cannot effectively represent customer characteristics in relation to purchasing behavior, and can effectively uncover customers' true preferences. After the missing information is generated, the original information of the text modal data of the supplemented customer feature information and the text modal data of the item attributes in S100 is converted into numerical embedding vectors using a pre-trained Transformer MiniLM model. Each piece of text information is mapped to a high-dimensional space. The customer's text information is represented by Equation 3, and the item attribute's text information is represented by Equation 4.

[0071] (3)

[0072] (4)

[0073] The raw information of the image modal data of the item attributes is converted into a numerical embedding vector using the CLIP model, as shown in Formula 5.

[0074] (5)

[0075] During the input phase, customer feature information is embedded into vectors instead of the original random customer features for recommendation input, and item features are embedded into vectors instead of item IDs. This allows for effective subsequent mining of the impact of customer features on purchasing behavior.

[0076] Step 300: Construct a customer-wine social hypergraph and quantify the weight of wine and customers in social relationships by integrating the global and local importance of wine and customers in the hypergraph.

[0077] To address the shortcomings of existing graph-based multimodal recommendation algorithms in uncovering social relationships—specifically, neglecting the influence of indirect social relationships between consumers and their direct and indirect friends, as well as the impact of implicit social relationships between purchased items—this invention innovatively constructs a hypergraph of customers and wine items to mine potential social relationships and quantifies node importance.

[0078] Given a set of customers C, c∈C, and a set of wine products W∈W. Model the binary relations of interactions as a customer-wine bipartite graph G = {C, W, E}, where historical interactions are considered as the set of edges in the graph denoted by E. In addition to interaction information, each item contains multimodal content information m∈{v, t}, where v and t represent image features and text features, respectively. Represent the modal features of a term w as... ,in Let v represent the feature dimension of mode m. In this study, only image and text modes are considered, represented by v and t. Therefore, the proposed framework can be easily extended to scenarios involving both modes.

[0079] Define a customer-wine hypergraph and denote it as... The nodes and edges of this hypergraph represent the complex interactions between customers and wine items, and the weights represent the strength of the relationships, taking into account customers' social relationships, personal characteristics, and item attributes.

[0080] in, The nodes in the data are divided into the following categories: given client nodes : Represents a set of m customers in the data. Each customer node Each node corresponds to a specific customer, connecting that customer's personal characteristics and purchase history. (Wine node) : Represents the set of wine items in the data, with a quantity of n. Each wine node Each corresponds to a type of wine, linked to its color, brand, aroma, and other attributes. Customer personal characteristic node ( ): Represents the individual characteristics of each customer, where This includes the customer's specific gender, age, marital status, occupation, monthly disposable income, and education level. (Wine attribute node) ): Represents the attributes of each wine. This includes the grapes used for winemaking, the wine name, taste, acidity, sparkling properties, sweetness, aroma, color, country, region, vintage, and ABV value.

[0081] It includes two forms: edges and superedges. This invention provides a client with... Give wine The interactive behavior, i.e., rating, is The rating edge between the customer and the wine is represented as... ,in Indicates customer Give wine The rating relationship is represented by this edge, which connects the customer node and the wine node, and carries rating values. ;client With the client's personal characteristics The edge between can be represented as ;thing With item attributes The edge between can be represented as .

[0082] Unlike traditional edges, hyperedges can connect more than two nodes. In customer relationship modeling, this embodiment of the invention constructs three types of hyperedges for different relationship types between customers: given customers with shared interests as friends, this type of social hyperedge with direct friend relationships is represented by the following formula:

[0083] (6)

[0084] in, It is a subset of customers who share common interests. The weight of the hyperedge is derived by calculating the number of wines that customers jointly rate; the social hyperedge with shared demographic characteristics among customers is represented by the following formula:

[0085] (7)

[0086] in, It is a subset of customers, and the customers within this subset share common demographic characteristics. The weight of a hyperedge is derived by calculating the number of shared demographic features among customers; the indirect social hyperedge with mutual friends in a social network is represented by the following formula:

[0087] (8)

[0088] in, It is a subset of customers, and the customers in the subset have mutual friends. The weight of the hyperedge is derived by calculating the number of common friends between customers. In the wine item relationship, three types of hyperedges are constructed between items. The social hyperedge of common attributes shared by wines is represented by the following formula:

[0089] (9)

[0090] in, This represents a subset of wine items, where the wines within the subset share common attributes. The weight of a hyperedge is the number of common attributes it shares; the social hyperedge of a purchased item pair is represented by the following formula:

[0091] (10)

[0092] in, A subset of wine items, where all wines in the subset are purchased simultaneously. The weight refers to the number of times a purchase is made simultaneously; the number of common customers making purchases on the social super-edge.

[0093] (11)

[0094] in, This represents a subset of wine items, where the wines in the subset were purchased by a common customer. The weight of the superedge is the number of identical customers.

[0095] Furthermore, this embodiment of the invention, based on the constructed customer-wine item hypergraph, calculates the global and local importance of each customer and item through social relationships. Global importance is calculated using the PageRank algorithm, and local importance is calculated using degree centrality. Wherein, given the hypergraph... All nodes in the middle are denoted as The edge set is PageRank is calculated for each customer through recursive relationships. Global importance of nodes and wine items Global importance of nodes This is represented by the formula:

[0096] (12)

[0097] in, The damping coefficient is 0.85, representing the probability of information propagation. It is the total number of nodes in the overrun. Indicates pointing to the customer The set of nodes, Indicates referring to wine The set of nodes, Represents a node The set of out-degrees.

[0098] client and wine items Local importance is measured by degree centrality, which reflects the number of direct connections a node has, denoted as: As shown in the formula, It is a node The degree.

[0099] (13)

[0100] Further combining global importance with local importance yields the importance of customers and items as shown in the formula:

[0101] (14)

[0102] Step 400: Based on the subgraphs with actual interactive behaviors extracted from the social hypergraph, construct a user-wine isomorphic graph to enhance the learning of positive customer behaviors.

[0103] The relationship between customers and wines is often represented by matrices (e.g., behavioral matrices such as ratings and collections). However, customers only engage with a small subset of wine items and have limited interaction with most others, leading to sparsity in the matrix and limiting its ability to fully capture the complex, multi-layered, and diverse interactions between customers and wines. Therefore, this invention only considers wines that have interacted with customers. Based on customer-item interaction behavior, it innovatively constructs a customer-wine item isomorphism graph to further optimize interaction effects and explicitly models customer wine preference patterns to enhance the capture of personalized customer preferences. This paper extracts customer-item interaction behavior (ratings) from a hypergraph to construct an isomorphism graph. ,in, , This refers to wine items that have interacted with customer C. Each edge represents a regularization of the customer's wine rating. For each customer Retain the highly rated wine items that the customer interacted with. Specifically, when customers... Interactive wine collection quantity Greater than At that time, if For wines that rank in the top-K of the ratings, the edge weights are retained. When the customer Number of interactive wine collections Less than At that time, it will randomly interact with... We select the quantity that satisfies K, as shown in the following formula.

[0104] (15)

[0105] After establishing the customer-item isomorphism graph, an attention mechanism was added during graph propagation. The weights used to aggregate user-adjacent nodes were calculated using the softmax function to maximize the impact of interacting wine item pairs on customers. Layered aggregation learns customer representation This can be expressed as the following formula:

[0106]

[0107] Step 500: Inject the captured social weights and relationships into the customer and wine embedding vectors, and realize the multimodal information fusion representation of customer preferences.

[0108] To fuse unimodal features from different modalities, multimodal feature fusion is optimized for both customers and items. For item multimodal embedding, the Attentive Concatenation method is used, which weights and fuses features from each modality based on user preferences for different modalities, paying attention to the weights. It was initialized to 0.5. The user's multimodal embeddings are then directly concatenated, simply piecing together the information from each modality. Among these, and A fusion of customer and wine items, and These represent the client's embedding representation in the image modality and text modality, respectively. and These represent the embedded representations of wine items in the visual and textual modalities, respectively. The text indicates a splicing operation. Here, the multimodal fusion representation for customers is weighted and spliced, while the multimodal fusion representation for wine items is spliced ​​directly. This approach better preserves the unique information of each modality while integrating complementary information from different modalities, thus improving recommendation accuracy.

[0109] (17)

[0110] Furthermore, the customer and wine representations learned from the graph, along with the customer and item importance weights computed from the customer-wine item hypergraph, are integrated to form their ensemble representation, which effectively captures the interactions between customers and wines and their underlying implicit relationships. The learned outputs are then summed element-wise to generate customer representations. and wine The dual representation of is shown in the following formula.

[0111] (18)

[0112] in, and Indicates customer and wine The final representation. and These represent the GCN layers of the customer item isomorphism graph and the project semantic graph, respectively.

[0113] Step 600: In the prediction phase, based on the subgraphs with actual interactive behaviors, positive feedback, negative feedback, and neutral feedback states are proposed to improve the BPR loss function, and the ranking results of the wines that customers may like are obtained for recommendation.

[0114] In the parameter optimization stage of the recommendation task, the Bayesian Personalized Ranking (BPR) loss optimizes the ranking pairs by learning the relative relationship between positive and negative samples, thereby improving the personalization and accuracy of the recommendation system. In practice, we have found that customer ratings of wines vary; higher ratings indicate greater customer satisfaction, while lower ratings suggest lower preferences after actual tasting. Therefore, this invention optimizes the Bayesian Personalized Ranking (BPR) loss function by reconstructing the loss function based on subgraphs with actual interaction behavior. The relationship between wines not rated by the customer and the customer is set to a negative state, and a threshold is set for wine ratings. Interactions exceeding a threshold are defined as positive states, and those below the threshold are defined as neutral states. Positive state sample pairs and negative state sample pairs, as well as neutral state samples and negative state samples, are generated respectively. The reconstruction loss function is defined as the formula, where... Calculate the inner product between the two. The inner product of customer characterization and wine characterization. The inner product of customer characterization and wine characterization with positive feedback. The inner product of customer characterization and wine characterization with neutral feedback. The inner product of customer characterization and wine characterization with negative feedback. The parameter representing the loss between neutral and negative samples. yes Regularization weights, This represents the model's parameters. The reconstructed loss function strengthens the learning of positive samples. The combination of the two losses allows the model to learn, through implicit feedback, how to rank wines that customers are interested in and those they are not, while also taking into account neutral samples, thereby further refining the prediction of customer preferences.

[0115] (19)

[0116] like Figure 2 As shown, this invention also provides a wine recommendation system based on multimodal heterogeneous data, the system comprising:

[0117] A missing information generation module, which generates missing wine attributes and customer characteristics based on a fine-tuned LLM;

[0118] The interaction-aware preference mining module, based on the customer-wine social hypergraph and the customer-wine isomorphic graph, injects the captured social weights and relationships into the customer and wine embedding vectors, and realizes the multimodal information fusion representation of customer preferences;

[0119] The prediction module optimizes the loss function based on the subgraphs with actual interactive behavior to obtain the final ranked wine recommendation results.

[0120] This embodiment optimizes the multimodal wine recommendation algorithm based on fine-tuned LLM and social interaction graphs, solving the difficulties of ignoring the impact of customer characteristics on purchasing behavior and ignoring the implicit social meanings under interaction behavior, thereby improving the accuracy of wine recommendation based on multimodal heterogeneous data.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A wine recommendation method based on multimodal heterogeneous data, characterized in that, Includes the following steps: Step S100: Construct missing information on wine attributes and customer characteristics based on a fine-tuned LLM for wine and its customers; Step S200: Based on the information generated by fine-tuning the LLM in S100, a pre-trained Transformer model is used to convert customer features and wine information from their original form into digital embedding vectors instead of random embedding vectors; the customer features and wine information include images and text. Step S300: Construct a customer-wine social hypergraph and quantify the weight of wine in social relationships by integrating the global and local importance of wine and customers in the hypergraph. Step S400: Extract subgraphs with actual interactive behaviors from the social hypergraph to construct a user-wine isomorphic graph to enhance the learning of positive customer behaviors; Step S500: The captured social weights and relationships are injected into the embedding vectors of customers and wines, and a multimodal information fusion representation of customer preferences is achieved. Step S600: In the prediction phase, based on the subgraph with actual interactive behavior, positive feedback, negative feedback, and neutral feedback states are proposed to improve the BPR loss function, and the ranking results of the wines that customers may like are obtained for recommendation.

2. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step S100 includes: retrieving an existing publicly available wine dataset; Based on the wine attribute information in the dataset 1 above, a wine attribute generation Prompt is constructed, and the Prompt is used to fine-tune the large-scale language model LLM to generate the missing wine attributes in the dataset. Based on the rating data and complete wine attribute data in the dataset 1 above, we can calculate the types of wine attributes that each customer prefers. Retrieve different datasets; calculate the similarity of wine attribute preferences between customers in dataset 2 and customers in dataset 1 based on cosine similarity, and filter the top k customers; Construct a Prompt based on the customer's personal characteristics, and use this Prompt to fine-tune a large-scale language model LLM; The finely tuned LLM, as a generative model, can effectively generate missing customer personal characteristics in Dataset 1, including gender, age, marital status, occupation, monthly disposable income, and education level, thereby enhancing customer information.

3. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step S200 includes: The MiniLM model, using a pre-trained Transformer, converts the raw information of the text modal data of customer feature information and the text modal data of wine attributes in S100 into numerical embedding vectors. The CLIP model is used to convert the raw information of image modal data of wine attributes into numerical embedding vector representations. During the input phase, the embedding vector of customer feature information is used to replace the random feature embedding of customers as the initial representation of customers, and the embedding vector of wine features is used to replace the item ID embedding vector in the wine attributes as the initial representation of wine.

4. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step S300 includes: Based on customer characteristics, wine attributes, and customer-wine interaction behavior, three types of edges and six types of hyperedges are defined to construct a customer-wine social hypergraph. Calculate the global and local importance of each customer and wine based on the social edges in the customer-wine social hypergraph; Combining global importance with local importance yields the social weight of customers and items.

5. The wine recommendation method based on multimodal heterogeneous data according to claim 4, characterized in that, The global importance is calculated using the PageRank algorithm, and the local importance is calculated using degree centrality.

6. The wine recommendation method based on multimodal heterogeneous data according to claim 4, characterized in that, The method of combining global importance and local importance to obtain the social importance of customers and items includes: A matrix list is generated by arranging the global and local importance of each customer and wine, which serves as the social weight of customers and items.

7. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step 400 includes: Based on customer interaction behavior with items, an isomorphic graph is constructed by extracting customer interaction behavior ratings from the customer-wine social hypergraph and setting an isomorphic graph threshold. If the number of wines interacted with by a customer is greater than the threshold, then the wines with the highest ratings are selected. If the number of wines interacted with by a customer is less than the isomorphic graph threshold, then the number of wines randomly selected from the interacted items is increased to reach the threshold, thus achieving the purpose of isomorphism.

8. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step S500 includes: fusing multimodal information on customers and wine respectively; The multimodal embedding of wine uses the Attentive Concatenation method, which weights and fuses features of each modality based on the user's preference for different modalities. Note that the weights are initialized to 0.

5. The customer's multimodal embedding is directly spliced ​​together, simply piecing together the information from each modality; An integrated representation of customers and wines is formed by learning representations of customers and wines from the customer-wine social hypergraph and the user-wine isomorphism graph, as well as calculating the importance of customers and items from the customer-wine social hypergraph.

9. The wine recommendation method based on multimodal heterogeneous data according to claim 1, characterized in that, Step S600 includes: Based on the interaction state, the relationship between the customer and the wine that has not been rated by the customer is set to a negative state. A threshold is set for wine rating. Interactions above the threshold are defined as positive states, and those below the threshold are defined as neutral states. Sample pairs are generated for each, and the difference in user preference ranking between positive and negative samples and between neutral and negative samples is maximized to obtain the optimized loss function.

10. A wine recommendation system based on multimodal heterogeneous data, characterized in that, include: A missing information generation module, which generates missing wine attributes and customer characteristics based on a fine-tuned LLM; The interaction perception preference mining module, based on the customer-wine social hypergraph and the user-wine isomorphic graph, injects the captured social weights and relationships into customer and wine nodes, and realizes the multimodal information fusion representation of customer preferences; The prediction module optimizes the loss function based on the customer's interaction with the wine to obtain the final ranked wine recommendation results.

Images