An NFT recommendation method based on a multi-modal graph attention model

Abstract

The application discloses an NFT recommendation method based on a multi-modal graph attention model, optimizes the problem of sparse user and NFT interaction data in an existing model, fully combines image, description text, transaction data and other information to extract the features of the user and the NFT, trains an attention network model, and obtains a recommendation result, and comprises the following steps: firstly, NFT image data, text data, price data and transaction data are acquired, initial embedding vectors of the user and the NFT are obtained through feature extraction under each data mode, then a graph model is constructed to obtain high-order relationships of the user and the NFT, a final user vector and an NFT vector are obtained by combining a gating mechanism and an attention mechanism, and a preference value of the user to a given NFT can be obtained through inner product of the two, and a recommendation result is obtained through sorting from high to low.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of recommendation algorithms, more particularly to an NFT recommendation method based on a multi-modal graph attention mechanism BACKGROUND

[0002] Many fields regard recommendation systems as an important tool to enhance user experience, such as movies, music, and e-commerce. In these fields, recommendation systems have been extensively studied to enable users to better find content they may like.

[0003] In recent years, the NFT (Non-Fungible Token) market has experienced rapid growth in size and transaction volume, demonstrating broad application prospects in sports, games, art collections, music, and other industries. NFT recommendation algorithms face numerous challenges in practical applications. First, NFTs are exclusive and can only be owned by one user at a time, and can only be transferred to another user by the current owner in a transaction. In contrast, online music, movies, and goods can be liked and shared by multiple users simultaneously, allowing for the collection of a large amount of feedback information. Second, NFT transactions are anonymous. NFTs run on a blockchain system, which is an anonymous system that uses encryption to allow users to transact without directly exposing their identity information. Therefore, it is difficult to obtain various user characteristics in terms of user dimension. Finally, users in the NFT market have diverse purchasing motivations. NFTs can be purchased as digital collectibles or purely speculative products, which makes it necessary for recommendation systems to consider not only the content attributes but also the asset attributes such as price, turnover frequency, and transaction parties, adding challenges to the architecture of the recommendation system.

[0004] Currently, the scene of recommendation systems suitable for NFTs is still relatively blank, and the unique characteristics of the NFT market and the increasingly prominent development prospects highlight the importance of developing personalized recommendation systems to meet specific needs. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an NFT recommendation method based on a multi-modal graph attention model.

[0006] The present application optimizes the problem of sparse user and NFT interaction data in existing models, fully combines image, description text, transaction data, and other information to extract user and NFT features, uses transaction data to construct a graph model of users and NFTs, obtains more rich information from the high-order connectivity between nodes, obtains the weights of the influence of nodes on each other according to the attention mechanism, and finally obtains the vector representation of users and NFTs, and the inner product of the two obtains the recommendation result.

[0007] The technical scheme adopted by the present application to achieve the above-mentioned purpose is as follows:

[0008] The NFT recommendation method based on the multi-modal attention model adopts a recommendation model based on a multi-modal attention model, and the model includes a data preprocessing module, a feature extraction module, a graph attention learning module, and a recommendation generation module. The method comprises the following steps:

[0009] Step S1: Obtain an NFT dataset, which includes NFT image data, transaction data, price data, and NFT description text data, and preprocess the data;

[0010] Step S2: Construct a convolutional neural network model to extract features from the NFT image data obtained in step S1, and obtain the feature representation of the NFT in the image modality;

[0011] Step S3: Construct a word embedding model to extract features from the NFT description text data obtained in step S1, and obtain the feature representation of the NFT in the text modality;

[0012] Step S4: Construct a neural network model to process the price data and transaction data obtained in step S1, and extract the feature representation of the user and the NFT in the transaction modality;

[0013] Step S5: Use the transaction data obtained in step 1 to construct a user and NFT interaction behavior matrix, use the NFT feature vectors obtained in steps S2 and S3 and the feature vectors of the user and the NFT in the transaction modality obtained in step S4, combine the gating mechanism and the attention mechanism, and define the aggregation formula of the nodes in the interaction graph;

[0014] Step S6: Information aggregation, use neighbor nodes to update the value of a specific node, aggregate the node representations in different modalities, then obtain a vector representation containing high-order connectivity information, and combine it into a final node vector.

[0015] Step S7: According to the vector representation of the user and the NFT obtained in step S6, perform an inner product operation on the two to predict a matching score, and sort the matching scores in descending order to obtain an NFT recommendation sequence for a specific user;

[0016] Step S8: Construct a loss function and use an AdamW optimizer to train the model.

[0017] As an optional technical solution, the step S2 specifically comprises:

[0018] Step S21: Adjust the image shape in the NFT image data set obtained in step S1 to a 256*256*3-dimensional vector.

[0019] Step S22: load the convolutional network model, input the picture to be processed into the model, obtain the vector representation of each picture through model training.

[0020] As an optional technical solution, the step S3 specifically comprises:

[0021] Step S31: process the step 1 Chinese text data set, and convert the text into a plurality of description words of the NFT through word segmentation and stop word processing;

[0022] Step S32: load the word vector model, convert the words in the text into fixed-dimensional word vectors, and then connect the plurality of words corresponding to each NFT, if the number of words is less than 6, use zero padding to fill the remaining values.

[0023] As an optional technical solution, the step S4 specifically comprises:

[0024] Step S41: calculate the average purchase price of each user to represent the financial ability and willingness to purchase NFT of the user;

[0025] Step S42: calculate the average time of the user holding NFT, in days, to represent the transaction frequency of the user in the NFT market, if the value is small, it indicates that the user is more likely to be a speculator rather than a collector in the NFT market;

[0026] Step S43: calculate the total transaction times of the user to represent the activity of the user in the market;

[0027] Step S44: establish a fully connected layer, the input is a vector composed of three values obtained in steps S41, S42 and S43, and the output of the fully connected layer is a user feature vector;

[0028] Step S45: calculate the average transaction price and transaction times of each NFT;

[0029] Step S46: establish a fully connected layer, the input is a vector composed of values obtained in step S45, and the output of the fully connected layer is a feature vector of the NFT in the transaction mode.

[0030] As an optional technical solution, the step S5 specifically comprises:

[0031] Step S51: construct an interaction matrix of users and NFTs to represent the implicit feedback information of users and NFTs, and the element in the matrix is 1, indicating that the corresponding user and NFT have interacted. Based on the interaction matrix, a multi-modal graph is constructed, including an image modal, a text modal and a transaction modal;

[0032] Step S52: initialize the node vector, set the user vector and the item vector as a dm-dimensional vector for the modality m. The user vector is passed from the full connection layer in step S44, and the NFT vector is obtained in steps S2, S3, S46;

[0033] Step S53: use the Gated Attention-GNN model, which combines the gating mechanism and the attention mechanism, to dynamically select and integrate the node's own features and neighbor node features using the relationship between the nodes, to obtain the user and item vectors fused with the neighbor nodes. Specifically, for a node h in the user-NFT interaction matrix, there are N h ={t | (h, t) E}, then the information of h can be represented by neighbor aggregation;

[0034] Step S54: under modality m, calculate the gating values: 1. inner product gate, calculate the inner product of the vector e m,h of the node h and the vector e m,t of its adjacent nodes, and then normalize. 2. connection gate, use linear transformation to connect e m,h and e m,t , multiply them with a trainable weight matrix, and then normalize. 3. double interaction gate, combine the outputs of the inner product gate and the connection gate, and fuse the similarity based on the inner product and the connection representation, to provide a more flexible and powerful mechanism for information propagation in the model;

[0035] Step S55: use the attention mechanism to learn the importance of each neighbor to the change of the node. Use the vector inner product to obtain the attention weight, which indicates the similarity between the two nodes, and then use the softmax function to normalize the weights of all neighbor nodes.

[0036] As an optional technical solution, the step S6 specifically includes:

[0037] Step S61: use the neighbor node to update the value of the node h

[0038]

[0039] where W m,2 is a transformation matrix, e h serves as a virtual super node connecting {e m,h}, and then combine and

[0040]

[0041] where, ​denotes the representation of node h after first-order connection encoding, W m,3 is a trainable weight matrix.

[0042] Step S62: Further represent the nodes by using high-order connectivity between the nodes, and the aggregation formula is defined as:

[0043]

[0044] wherein, is the result after the l-1th propagation, which stores the information from the lth order, is the initial embedding vector e m,h After updating the representation of the nodes in a specific modality m, the representations of different modalities are combined into a new representation according to the average;

[0045] Step S63: The multi-order vectors of the user and the NFT are spliced to obtain the final representation of each, and the order is 3.

[0046] As an optional technical solution, the step S8 specifically comprises:

[0047] Optimize the model parameters using Bayesian personalized recommendation, as follows

[0048]

[0049] wherein, O∈{(u,i,j)|(u,i)∈R + ,(u,j)∈R -} represents the training set, R + represents a data set containing observed interactions between the user u and the item i, and R - represents the unobserved. δ(.) is a sigmoid function, λ is a decay factor, and θ is a parameter used in the model.

[0050] The working principle of the present application is:

[0051] Firstly, the present application fully utilizes the information of the NFT recommendation market, obtains the data of the NFT and the user under different modalities, and weakens the influence of data sparsity on the effect of the recommendation model; secondly, the present application extracts high-order interaction information of the user and the NFT by using the properties of the interaction graph, and can obtain more extensive recommended content; finally, the present application uses the attention mechanism and the gating mechanism to obtain the influence weight of different modal data on a given node and the influence weight of different nodes on the given node, so that the recommendation effect is more accurate.

[0052] The present application has the following advantages:

[0053] The application fully utilizes various data of the NFT recommendation market, constructs a recommendation model based on a multi-modal graph attention mechanism, and achieves better results than a benchmark model under a recommended algorithm-based measurement standard. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a model structure diagram of the method of the application.

[0055] Figure 2 is a flowchart of the method of the application. DETAILED DESCRIPTION

[0056] The technical solutions of the application are further described below with reference to the drawings.

[0057] The embodiment relates to an NFT recommendation method based on a multi-modal graph attention model, optimizes the problem of sparse user and NFT interaction data in an existing model, fully combines image, description text, transaction data and other information to extract features of users and NFTs, trains an attention network model, and obtains a recommendation result. The application comprises the following steps: firstly, NFT image data, text data, price data, transaction data and user data are acquired, initial embedding vectors of the respective data are obtained through feature extraction, a graph model is then constructed to obtain more accurate vector representations of users and NFTs, and finally, user vectors and NFT vectors are obtained through an attention network model, and a preference value of a user to a given NFT can be obtained through dot product of the two vectors, and a recommendation result can be obtained through sorting from high to low.

[0058] As Figure 1 , Figure 2 , the specific steps of the application comprise:

[0059] Step S1: In this example, several most popular NFT sets in the opensea NFT transaction website are selected for implementation, including 500,000 transaction data, 24,000 NFT images and 24,000 NFT text description information, which are concentrated in the field of art collections. The NFT text description exists in the form of multiple key-value pairs, such as {Hat: 'M1 Beanie'; Mouth: 'M1 Bored'}. The transaction types of the transaction data include transfer and sale, and a transaction can be described by a four-tuple (u i ,u j ,k,p), which represents that the user u i trades the NFT k to the user u j , and p represents the transaction price, if the transaction type is transfer, then p=0. In order to avoid that the training data is too sparse, the users and NFTs with interaction data greater than 5 times are intercepted, and 80% of them are used as a training set and 20% of them are used as a test set.

[0060] Step S2: First, the image shape in the NFT image dataset obtained in step 1 is adjusted to a 256*256*3-dimensional vector. A pre-trained CAE (Convolutional Auto-Encoder) model is loaded, which includes an encoder and a decoder, both composed of 8 fully connected layers. The encoder uses a 3*3 convolution kernel and a 2*2 max pooling kernel, while the decoder uses a 3*3 convolution kernel and a 2*2 upsampling. All nonlinear functions in the model are implemented by ReLU activation functions. The pictures to be processed are input into the CAE model, and the vector representation of each picture is obtained through model training.

[0061] Step S3: Process the text dataset in step 1, and convert the text into multiple descriptive words for NFT. Load the pre-trained Glove model to convert the words in the text into 300-dimensional word vectors, and then connect the multiple word embeddings corresponding to each NFT. If the number of words is less than 6, use zero padding to fill the remaining values.

[0062] Step S4: Process the transaction dataset in step S1, extract the user's feature vector, such as steps S41 to S44; obtain the feature vector of NFT, such as steps S45 to S46;

[0063] Step S41: Calculate the average purchase price of each user to represent the user's financial ability and willingness to purchase NFT;

[0064] Step S42: Calculate the average time of user holding NFT in days to represent the user's trading frequency in the NFT market. If the value is small, it indicates that the user is more likely to be a speculator rather than a collector in the NFT market

[0065] Step S43: Calculate the total number of user transactions to represent the user's activity in the market;

[0066] Step S44: Establish a fully connected layer, the input is a 1*3-dimensional vector composed of three values obtained from S41, S42 and S43, and the output is obtained after passing through 2 fully connected layers to get the user feature vector;

[0067] Step S45: Calculate the average transaction price and transaction frequency of each NFT;

[0068] Step S46: Establish a fully connected layer, the input is a vector composed of values obtained from step S45, and the output is obtained after passing through a fully connected layer to get the feature vector of NFT under the transaction mode.

[0069] Step S5: Construct an interaction graph between users and NFTs in a multi-modal manner, including the following steps:

[0070] Step S51: Let U denote the set of users, I denote the set of NFTs, and Y∈|U|×|I| denote the interaction matrix of users and NFTs, where y ui =1 indicates that user u interacts with NFT i, and this interaction includes transfer and purchase, otherwise 0. According to Y, a multi-modal graph G m can be constructed, where belongs to the set of modalities M = {image, text, transaction};

[0071] Step S52: Initialize node vectors, for each modality m, set the user vector and item vector as a dm-dimensional vector. The user vector is obtained by passing the fully connected layer in step S44, and the NFT vector is obtained in steps S2, S3 and S4.6;

[0072] Step S53: Construct a Gated Attention-GNN model that combines the gating mechanism and attention mechanism, uses the relationship between nodes, dynamically selects and integrates the node's own features and neighbor node features, and obtains user and item vectors that integrate neighbor nodes. Specifically, for a node h in Y, there are N h ={t | (h, t) ∈ E}, then the information of h can be aggregated from neighbors as

[0073]

[0074] where m denotes the modality, f a (h, t) and f g (h, t) are components for propagating gates and attention in the gated attention network, f g (h, t) is a propagation gate that determines whether information is propagated from t to h, and f a (h, t) is an attention score that represents the contribution value of t, W m,1 is a trainable weight matrix;

[0075] Step S54: Calculate the gating value between nodes, which is realized by the following process:

[0076] 1. Inner product gate, calculate the inner product of e m,h and e m,t , then is used for normalization

[0077]

[0078] 2. Connection gate, use linear transformation to connect e m,h and e m,t , || is a connection operator, W c is a trainable weight

[0079] matrix

[0080]

[0081] 3. Double-interaction gate, combining inner-product gate and connection gate

[0082]

[0083] This gate combines the outputs of inner-product gate and connection gate, while integrating both inner-product-based similarity and connection representation. It provides a more flexible and powerful mechanism for information propagation in the model.

[0084] Step S55: Learn the importance of each neighbor to the node change using attention mechanism, as follows:

[0085]

[0086] where tanh is a nonlinear activation function, W m,h and W m,t are learnable transformation matrices. Inner product is used to obtain attention weights, indicating the similarity of two nodes. Then, the weights of all neighbor nodes are normalized using the softmax function, as follows

[0087]

[0088] Step S6: Information aggregation, using neighbor nodes to update the value of node h,

[0089]

[0090] where W m,2 is a transformation matrix, e h is a virtual super node acting as a connection {e m,h}, After that, combine and

[0091]

[0092] where represents the representation of node h after first-order connection encoding, W m,3 is a trainable weight matrix.

[0093] Further representation of nodes using high-order connectivity between nodes, node h is defined as:

[0094]

[0095] where is the result after l-1 propagation, storing information from l-order, ​is the initial embedding vector e m,h After updating the representation of nodes in a specific modality m, the representations of different modalities are combined into a new representation as:

[0096]

[0097] The number of information propagation is L (L=3), and the final representation of the node is generated in the following way

[0098]

[0099] Step 7: Perform inner product operation on the vector representation of the user and the NFT to predict the matching score

[0100]

[0101] Step 8: Use Bayesian personalized recommendation to optimize model parameters as follows:

[0102]

[0103] Where O is {(u, i, j) | (u, i) is R + , (u, j) is R -} represents the training set, R + represents the data set containing the observed interaction between user u and item i, and R - represents the unobserved. δ(.) is the sigmoid function, and lambda is the decay factor. Theta is the parameter used in the model.

[0104] Step S9: Use the commonly used evaluation indicators Recall@30 and NDCG@30 in the recommendation system to obtain the recommendation effect comparison between the present method and the benchmark method, as shown in Table 1

[0105] Table 1

[0106] Model Recall@30 NDCG@30 LightGCN 0.152 0.061 DeepFM 0.157 0.053 WideDeep 0.185 0.057 MGAT 0.241 0.071 DCN 0.195 0.065 BPR 0.174 0.078 MAGM 0.315 0.094

[0107] As can be seen from the table, the present method (MAGM: Multi-Attention Graph Model) is significantly better than the existing method in the evaluation standard. This proves that the present method makes full use of the data of each modality of the user and the NFT, and uses mechanisms such as attention to obtain more accurate vector representations, so that the prediction result is more accurate.

[0108] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be considered as limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A multi-modal graph attention model-based NFT recommendation method, characterized in that, First, the NFT image data, text data, price data, and transaction data are obtained, and the initial embedding vectors of the user and NFT are obtained through feature extraction in each data modality. Then, a graph model is constructed to obtain the high-order relationship between the user and the NFT. Finally, the user vector and the NFT vector are obtained by combining the gating mechanism and the attention mechanism, and the inner product of the two vectors is the preference value of the user for the given NFT. The recommendation result is obtained by sorting the preference value from high to low. The method comprises the following steps: Step S1: obtaining an NFT dataset, the dataset comprising NFT image data, transaction data, price data, and NFT description text data, and preprocessing the data; Step S2: constructing a convolutional neural network model to extract features of the NFT image data obtained in step S1, and obtaining the feature representation of the NFT in the image modality; Step S3: constructing a word embedding model to extract features of the NFT description text data obtained in step S1, and obtaining the feature representation of the NFT in the text modality; Step S4: constructing a neural network model to process the price data and the transaction data obtained in step S1, and extracting the feature representation of the user and the NFT in the transaction modality; Step S5: constructing an interaction matrix of the user and the NFT using the transaction data obtained in step 1, combining the gating mechanism and the attention mechanism, and defining the aggregation formula of the nodes in the multi-modal graph using the NFT feature vectors obtained in steps S2 and S3 and the feature vectors of the user and the NFT in the transaction modality obtained in step S4; and constructing a multi-modal graph based on the interaction matrix, including the image modality, the text modality, and the transaction modality; Step S6: information aggregation, using neighbor nodes to update the value of the node, aggregating the node representations in different modalities, obtaining the vector representation containing high-order connectivity information, and combining into the final node vector; Step S7: performing an inner product operation on the vector representations of the user and the NFT obtained in step S6 to predict the matching score, and obtaining the NFT recommendation sequence of the user in descending order of the matching score; Step S8: constructing a loss function, and training the model using an AdamW optimizer.

2. The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, Step S2 specifically comprises: Step S21: adjusting the shape of the image in the NFT image dataset obtained in step S1 to a 256*256*3-dimensional vector; Step S22: loading a convolutional network model, inputting the picture to be processed into the model, and obtaining the vector representation of each picture through model training. 3.The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, Step S3 specifically comprises: Step S31: processing the text data set in step 1, performing word segmentation and stop word processing, and converting the text into a plurality of description words of the NFT; Step S32: loading a word vector model, converting the words in the text into fixed-dimensional word vectors, and then connecting the plurality of word embeddings corresponding to each NFT. If the number of words is less than 6, the remaining values are padded with zeros. 4.The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, Step S4 specifically comprises: Step S41: calculating the average purchase price of each user to represent the financial ability and willingness to purchase NFTs of the user; Step S42: Calculate the average time that the user holds the NFT, in days, to represent the user's trading frequency in the NFT market. If the value is small, it indicates that the user is more likely to be a speculator rather than a collector in the NFT market; Step S43: Calculate the total number of transactions of the user to represent the user's activity in the market; Step S44: Establish a fully connected layer, the input is a vector composed of three values obtained in steps S41, S42 and S43, and the output of the fully connected layer is the user feature vector; Step S45: Calculate the average transaction price and transaction times of each NFT; Step S46: Establish a fully connected layer, the input is a vector composed of values obtained in step S45, and the output of the fully connected layer is the feature vector of the NFT under the transaction mode.

5. The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, Step S5 specifically includes: Step S51: Construct an interaction matrix of users and NFTs to represent the implicit feedback information of users and NFTs. The element in the matrix is 1, indicating that the corresponding user and NFT have interacted; Step S52: Initialize the node vector. For mode m, set the user vector and the item vector to dm-dimensional vectors. The user vector is obtained by passing the fully connected layer in step S44, and the NFT vector is obtained in steps S2, S3 and S46; Step S53: use the Gated Attention-GNN model which combines the gating mechanism and attention mechanism, uses the relationship of nodes, dynamically selects and integrates the node itself features and neighbor node features, and obtains the user and item vectors fused with neighbor nodes; specifically, for the node h in the user-NFT interaction matrix, there are directly connected nodes N h ={t | (h, t) E}, the information of h can be represented by neighbor aggregation; Step S54: In mode m, calculate each gating value:

1. Inner product gate, calculate the vector e of node h. m,h and its neighboring node vector e m,t 1. Calculate the inner product of the product and then normalize it; 2. Use a connection gate to connect e using a linear transformation. m,h and e m,t 1. Multiply with the trainable weight matrix and then normalize; 2. Double interaction gate, which combines the inner product gate and the connection gate, combines the outputs of the two, and integrates the similarity and connection representation based on the inner product, providing a more flexible and powerful mechanism for information propagation in the model; Step S55: Use the attention mechanism to learn the importance of each neighbor to the node change. Use the inner product of vectors to obtain the attention weight, which indicates the similarity of the two nodes, and then use the softmax function to normalize the weights of all neighbor nodes.

6. The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, Step S6 specifically includes: Step S61: Update the value of node h using the neighbor nodes h = h + sum (neighbor nodes) where W m,2 is a transformation matrix, e h is a virtual supernode acting as a connection point, after which and are combined wherein, represents the representation of node h after first-order connection encoding, W m,3 is a trainable weight matrix; Step S62: Use the high-order connectivity between nodes to further represent the nodes. The aggregation formula is defined as: wherein, is the result after the l-1th propagation, storing information from the lth order, is the initial embedding vector e m,h ; after updating the representation of the node in the modalities m, the representations of the different modalities are combined into a new representation according to an average; Step S63: Concatenate the multi-order vectors of users and NFTs respectively to obtain the final representation of each.

7. The NFT recommendation method of a multi-modal graph attention model according to claim 1, wherein, In step S8, Bayesian personalized recommendation is used to optimize the model parameters where O ∈ {(u, i, j) | (u, i) ∈ R + , (u, j) ∈ R -} denotes the training set, R + denotes the set of observed interactions between user u and item i, and R - denotes the unobserved; δ(.) is the sigmoid function, λ is the decay factor, and θ are the parameters used in the model.

Images