Cold Start Optimization Method for Long-Sequence Recommendation Systems Based on Transformer

Abstract

This invention proposes a cold-start optimization method for long-sequence recommendation systems based on Transformer, belonging to the field of e-commerce recommendation technology. The method includes: collecting multi-source heterogeneous data, inputting it into a meta-learning model with a multilayer perceptron structure, and outputting interaction probabilities; training the model based on the cross-entropy loss function and stochastic gradient descent method to obtain the weights of user registration information data vectors and knowledge graph relationship vectors, constructing a cold-start user embedding vector; inputting this vector into a Transformer model to output recommended products, thus achieving accurate recommendations.

Description

Technical Field

[0001] This invention belongs to the field of e-commerce recommendation technology, specifically relating to a cold start optimization method for long sequence recommendation systems based on Transformer. Background Technology

[0002] In the digital age, the rapid development of the internet has led to an explosive growth in information. In e-commerce, recommendation systems can recommend products that users may be interested in based on their browsing and purchasing history, as well as behavioral data from other users. The Transformer, a commonly used model in recommendation systems, is a neural network model that includes components such as multi-head self-attention, feedforward networks, layer normalization, and residual connections. It can capture cross-modal relationships between users and items, ensuring the accuracy of recommendations.

[0003] However, for new users experiencing a cold start—that is, when a new user registers and uses the recommendation system for the first time—the Transformer system lacks historical behavioral data and cannot accurately grasp their interests and preferences, making it difficult to provide precise recommendations. For example, a new user registering on an e-commerce platform has no browsing or purchase history. The platform's recommendation system can only make recommendations based on some general popular products or popular preferences. Such recommendations often differ significantly from the new user's actual needs, resulting in a poor user experience.

[0004] Therefore, this invention proposes a cold start optimization method for long sequence recommendation systems based on Transformer, aiming to provide suitable vectors for the Transformer model during user cold start, so that it can provide accurate product recommendations for new users. Summary of the Invention

[0005] This invention proposes a cold start optimization method for long sequence recommendation systems based on Transformer, which enables the Transformer model to provide accurate product recommendations for users during cold starts.

[0006] To achieve the above objectives, the present invention proposes the following technical content:

[0007] A cold start optimization method for long-sequence recommendation systems based on Transformer includes the following steps:

[0008] S1: Collect multi-source heterogeneous data on different items from different fields and user groups; assume there are N multi-source heterogeneous data points in total.

[0009] S2: Transfer the i-th multi-source heterogeneous data e i The input is fed into the meta-learning model of the multilayer perceptron structure, and the meta-learning model of the multilayer perceptron structure is used to output the interaction probability in the i-th multi-source heterogeneous data.

[0010] S3: Based on the cross-entropy loss function, calculate the deviation L between the interaction probability and the actual interaction in all multi-source heterogeneous data. Based on the deviation L, use the stochastic gradient descent method to update the weights of the hidden layer and output layer of the meta-learning model of the multilayer perceptron structure to obtain the trained meta-learning model of the multilayer perceptron structure.

[0011] S4: The meta-learning model with a trained multilayer perceptron structure outputs the weights of the user registration information data vector and the weights of the knowledge graph relation vector, and establishes the embedding vector of the cold-start user;

[0012] S5: Input the embedding vector into the existing Transformer model, and the Transformer model outputs recommended products.

[0013] Furthermore, in step S4, the formula for the embedding vector is:

[0014]

[0015] In equation (1), w A The weights of the user registration information data vector are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3; w K The weights representing the relation vectors in the knowledge graph are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3; and These represent the normalized vectors of user registration information data and knowledge graph relation vectors for users starting from scratch.

[0016] Furthermore, assume that user e is cold-starting. A，新 and e K，新 They are respectively:

[0017] e A,新 =[e A,1 e A,2 ... e A,z ... e A,m ]

[0018] e K,新 =[e K,1 e K,2 ... e K,x ... e K,n ]

[0019] e A，新 and e K，新 Normalized and They are respectively:

[0020]

[0021]

[0022] For matrix e A，新 The z-th element e A，z After normalization, we get The corresponding element The formula is:

[0023]

[0024] In equation (2), Representation matrix The z-th element; e A,z Represents matrix e A，新 The z-th element; μ A,新 Represents matrix e A，新 The mean of all elements in the matrix; σ A,新 Represents matrix e A，新 The standard deviation of all elements in the sample;

[0025] Matrix e K，新 The x-th element e K，x After normalization, we obtain the matrix. The corresponding element The formula is:

[0026]

[0027] In equation (3), Representation matrix The x-th element; e K,x Represents matrix e K，新 The x-th element; μ K,新 Represents matrix e K，新 The mean of all elements in the matrix; σ K,新 Represents matrix e K，新 The standard deviation of all elements in the matrix.

[0028] Furthermore, in step S2, the hidden layer of the meta-learning model of the multilayer perceptron structure processes the i-th multi-source heterogeneous data e. i The calculation process is as follows:

[0029] The first computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture:

[0030] h1 = ReLu(W hid ·e i +b hid )

[0031] The second computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture:

[0032] h2=ReLu(W hid ·h1+b hid )

[0033] The output of the Kth computation of the hidden layer in the meta-learning model of the multilayer perceptron structure is as follows:

[0034] h K =ReLu(W hid ·h K-1 +b hid (4)

[0035] In equation (2), h K W represents the output of the meta-learning model of the multilayer perceptron architecture for the i-th hidden layer of multi-source heterogeneous data; ReLU(*) represents the ReLU activation function; W hid Represents the weight matrix of the hidden layer; e i b represents the i-th multi-source heterogeneous data; hid This represents the bias matrix of the hidden layer;

[0036] Output of the meta-learning model output layer in a multilayer perceptron architecture:

[0037]

[0038] In equation (5), represents the probability predicted by the meta-learning model of a user interacting with a corresponding item in the i-th multi-source heterogeneous data; Sigmoid(*) represents the Sigmoid activation function; W out h represents the weight matrix of the output layer. K b represents the output of the meta-learning model of the multilayer perceptron architecture for the i-th hidden layer of multi-source heterogeneous data; out This represents the bias matrix of the output layer.

[0039] Furthermore, the formula for the cross-entropy loss function in step S3 is:

[0040]

[0041] In equation (4), y i In real-world scenarios, 1 represents whether a user interacts with an item in the i-th multi-source heterogeneous data set; 0 represents the number of multi-source heterogeneous data sets if there is no interaction.

[0042] Furthermore, in step S3,

[0043] The update formulas for the weights of the hidden layer and the output layer are as follows:

[0044]

[0045] In equation (7), Wt+1 W represents the weight matrix of the hidden layer or output layer at the (t+1)th iteration. t This represents the weight matrix of the hidden layer or output layer at the t-th iteration; η represents the learning rate. This is the gradient of the loss function with respect to the weights of the hidden or output layer; the weight matrices of the hidden and output layers are continuously iterated during training.

[0046] Furthermore, when L is less than 0.05, it indicates that the meta-learning model of the multilayer perceptron structure has been trained.

[0047] The beneficial effects that can be achieved by adopting the above technologies are:

[0048] This solution first trains a meta-learning model with a multi-layer perceptron structure using multi-source heterogeneous data from non-cold-start users. The trained meta-learning model outputs weights for user registration information data vectors and knowledge graph relation vectors. During a user cold start, these two weights are fused together with the user registration information data vector and knowledge graph relation vector of the cold-start user to form an embedding vector. This embedding vector is then input into an existing Transformer model, which provides the final recommended products. Because it has been learned by the meta-learning model with a multi-layer perceptron structure, the output probability of the recommended products is more accurate. Attached Figure Description

[0049] Figure 1 This is the overall flowchart of this method;

[0050] Figure 2 This is a comparison chart of the recommendation accuracy using the Transformer model directly and the recommendation accuracy of the Transformer model after using the meta-model in this solution;

[0051] Figure 3 This is a comparison chart showing the recommendation probabilities of two methods for different products for a single type of user. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, the cold start optimization method for long sequence recommendation systems based on Transformer includes the following steps:

[0054] S1: Collect multi-source heterogeneous data on different items from different fields and user groups; multi-source heterogeneous data includes: user registration information data vector e A Knowledge graph relation vector e K User behavior feature vector e B Item attribute feature vector e F .

[0055] User registration information data vector e A Including: gender, age, region, etc.;

[0056] Knowledge graph relation vector e K This includes information such as interests and abilities.

[0057] User behavior feature vector e B This includes: clicks, browsing, and purchases.

[0058] Item attribute feature vector e F This includes: category, brand, price, etc.

[0059] Multi-source heterogeneous data e = [e A e B e K e F ];

[0060] For example:

[0061] The i-th multi-source heterogeneous data e i =[Male, 23; Loves football, buys football];

[0062] The (i+1)th multi-source heterogeneous data e i+1 =[Female, 26; enjoys watching dramas, clicking, phone stand];

[0063] The (i+2)th multi-source heterogeneous data e i+2 =[Female, 28; Likes basketball, has not purchased a basketball]. In this example, the multi-source heterogeneous data is displayed as text, while in reality it is displayed as vector numerical data.

[0064] S2: Transfer the i-th multi-source heterogeneous data e i The input is fed into the meta-learning model of the multilayer perceptron structure, and the meta-learning model of the multilayer perceptron structure outputs the interaction probability in the i-th multi-source heterogeneous data.

[0065] Specifically, the i-th multi-source heterogeneous data e i The input is fed into the hidden layer of the meta-learning model of the multilayer perceptron structure:

[0066] The first computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture:

[0067] h1 = ReLu(W hid ·e i +b hid )

[0068] The second computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture:

[0069] h2=ReLu(W hid ·h1+b hid )

[0070] The output of the Kth computation of the hidden layer in the meta-learning model of the multilayer perceptron structure is as follows:

[0071] h K =ReLu(W hid ·h K-1 +b hid (1)

[0072] In equation (1), h K W represents the output of the hidden layer of the meta-learning model of the multilayer perceptron architecture for the i-th multi-source heterogeneous data; ReLU(*) represents the ReLU activation function. hid Represents the weight matrix of the hidden layer; e i b represents the i-th multi-source heterogeneous data; hid This represents the bias matrix of the hidden layer.

[0073] Output of the meta-learning model output layer in a multilayer perceptron architecture:

[0074]

[0075] In equation (2), represents the probability predicted by the meta-learning model of a user interacting with a corresponding item in the i-th multi-source heterogeneous data; Sigmoid(*) represents the Sigmoid activation function; W out h represents the weight matrix of the output layer. K b represents the output of the meta-learning model of the multilayer perceptron architecture for the i-th hidden layer of multi-source heterogeneous data; out This represents the bias matrix of the output layer.

[0076] S3: Based on the cross-entropy loss function, calculate the deviation L between the interaction probability and the actual interaction in all multi-source heterogeneous data. Based on the deviation L, use stochastic gradient descent to update the weights of the hidden layer and output layer of the meta-learning model of the multilayer perceptron structure to obtain the trained meta-learning model of the multilayer perceptron structure.

[0077] The formula is:

[0078]

[0079] In equation (3), y i In real-world scenarios, this indicates whether a user interacts with an item in the i-th multi-source heterogeneous data set; 1 represents interaction, and 0 represents no interaction. N represents the number of multi-source heterogeneous data sets. The probability of a user interacting with a corresponding item in the i-th multi-source heterogeneous data, as predicted by the meta-learning model, is obtained from equation (2).

[0080] The update formulas for the hidden layer and output layer weights of the meta-learning model in a multilayer perceptron structure are as follows:

[0081]

[0082] In equation (4), W t+1 W represents the weight matrix of the hidden layer or output layer at the (t+1)th iteration. t This represents the weight matrix of the hidden layer or output layer at the t-th iteration; η represents the learning rate. This is the gradient of the loss function with respect to the weights of the hidden or output layer; the weight matrices of the hidden and output layers are continuously iterated during training.

[0083] After training through S2-S3, when L is less than 0.05, it indicates that the meta-learning model of the multilayer perceptron structure has been trained, and a well-trained meta-learning model of the multilayer perceptron structure has been obtained.

[0084] S4: The meta-learning model with a trained multilayer perceptron structure outputs the weights of the user registration information data vector and the weights of the knowledge graph relation vector, and establishes the embedding vector of the cold-start user.

[0085] When a cold-start user appears, since the user has no interactive behavior and no item preferences, they do not have e B，新 and e F，新 Therefore, it is necessary to perform a cold start on the user's e A，新 and e K，新 Establish the embedding vector for cold-start users. The formula for the embedding vector is:

[0086]

[0087] In equation (5), w A The weights of the user registration information data vector are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3; w K The weights representing the relation vectors in the knowledge graph are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3; and These represent the normalized vectors of user registration information data and knowledge graph relation vectors for users starting from scratch.

[0088] Assuming user e is cold-starting A，新 and e K，新 They are respectively:

[0089] e A,新 =[e A,1 e A,2 ... e A,z ... e A,m ]

[0090] e K,新 =[e K,1 e K,2 ... e K,x ... e K,n ]

[0091] e A，新 and e K，新 Normalized and They are respectively:

[0092]

[0093] For matrix e A，新 The z-th element e A，z After normalization, we get The corresponding element The formula is:

[0094]

[0095] In equation (6), Representation matrix The z-th element; e A,z Represents matrix e A，新 The z-th element; μ A,新 Represents matrix e A，新 The mean of all elements in the matrix; σ A,新 Represents matrix e A，新 The standard deviation of all elements in the matrix.

[0096] Matrix e K，新 The x-th element e K，x After normalization, we obtain the matrix. The corresponding element The formula is:

[0097]

[0098] In equation (7), Representation matrix The x-th element; e K,x Represents matrix e K，新 The x-th element; μK,新 Represents matrix e K，新 The mean of all elements in the matrix; σ K,新 Represents matrix e K，新 The standard deviation of all elements in the matrix.

[0099] S5: Input the embedding vector from step S4 into the Transformer model, and the Transformer model outputs recommended products.

[0100] Since step S4 optimizes the user registration information data vector and knowledge graph relationship vector of the cold start user using the corresponding weights of the trained multilayer perceptron meta-learning model, the Transformer model can more accurately recommend products after capturing the above features.

[0101] Calculation example 1:

[0102] From the new user group, 10,000 test users were randomly selected and divided into typical groups according to age and gender. Result 1 was obtained by directly using the Transformer model to predict the data; Result 2 was obtained by using the meta-learning model with a pre-trained multilayer perceptron structure to generate embedding vectors and then using the Transformer model to predict the data. The accuracy of the relevant data is shown in Table 1, and the results are as follows: Figure 2 .

[0103] Table 1. Accuracy results of recommended products for different users

[0104]

[0105] As can be seen from Table 1, this approach improves accuracy compared to directly using the Transformer model for prediction.

[0106] Calculation example 2:

[0107] For a single typical user (e.g., male, 23 years old, knowledge graph tag: loves football), the Transformer model directly predicts the probability of this user interacting with the product, denoted as Result 3. This approach denotes the probability of this user interacting with the product as Result 4. The accuracy of the relevant data is shown in Table 2; the corresponding results are shown in... Figure 3 .

[0108] Table 2 Probability of a Single Typical User Recommending a Product

[0109] 1 smartphones 0.35 0.03 5 Bluetooth headsets 0.28 0.12 37 Professional football boots 0.26 0.63 42 Heart rate fitness tracker 0.03 0.15 89 Team souvenirs 0.01 0.06 205 Phone stand 0.07 0.01

[0110] As can be seen from Table 2, this solution can accurately capture the user's knowledge graph information. For users whose vectors are related to football, "professional football shoes" has the highest recommendation probability, thus accurately recommending the items that users should be most interested in.

[0111] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A cold start optimization method for long-sequence recommendation systems based on Transformer, characterized in that, Includes the following steps: S1: Collect multi-source heterogeneous data on different items from different fields and user groups; There are N heterogeneous data sources; the heterogeneous data sources include: user registration information data vectors. e A Knowledge graph relation vectors e K User behavior feature vector e B Item attribute feature vector e F ; S2: Transfer the i-th multi-source heterogeneous data e i The input is fed into the meta-learning model of the multilayer perceptron structure, and the meta-learning model of the multilayer perceptron structure is used to output the interaction probability in the i-th multi-source heterogeneous data. S3: Based on the cross-entropy loss function, calculate the deviation L between the interaction probability and the actual interaction in all multi-source heterogeneous data. Based on the deviation L, use the stochastic gradient descent method to update the weights of the hidden layer and output layer of the meta-learning model of the multilayer perceptron structure to obtain the trained meta-learning model of the multilayer perceptron structure. S4: The meta-learning model with a trained multilayer perceptron structure outputs the weights of the user registration information data vector and the weights of the knowledge graph relation vector, and establishes the embedding vector of the cold-start user; S5: Input the embedding vector into the existing Transformer model, and the Transformer model outputs recommended products; In step S2, the hidden layer of the meta-learning model of the multilayer perceptron structure processes the i-th multi-source heterogeneous data e. i The calculation process is as follows: The first computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture: ； The second computation of the hidden layer in the meta-learning model of the multilayer perceptron architecture: ； The output of the Kth computation of the hidden layer in the meta-learning model of the multilayer perceptron structure is as follows: (4)； In equation (4), h K W represents the output of the meta-learning model of the multilayer perceptron architecture for the i-th hidden layer of multi-source heterogeneous data; ReLU(*) represents the ReLU activation function; W hid Represents the weight matrix of the hidden layer; e i b represents the i-th multi-source heterogeneous data; hid This represents the bias matrix of the hidden layer; Output of the meta-learning model output layer in a multilayer perceptron architecture: (5)； In equation (5), represents the probability predicted by the meta-learning model of a user interacting with a corresponding item in the i-th multi-source heterogeneous data; Sigmoid(*) represents the Sigmoid activation function; W out h represents the weight matrix of the output layer. K b represents the output of the meta-learning model of the multilayer perceptron architecture for the i-th hidden layer of multi-source heterogeneous data; out This represents the bias matrix of the output layer; In step S4, the formula for the embedding vector is: (1)； In equation (1), The weights of the user registration information data vector are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3. The weights representing the relation vectors in the knowledge graph are obtained from the output of the meta-learning model of the pre-trained multilayer perceptron structure in S3; and These represent the normalized vectors of user registration information data and knowledge graph relation vectors for users starting from scratch.

2. The cold start optimization method for long-sequence recommendation systems based on Transformer according to claim 1, characterized in that, Configure the user's e for cold start A，新 and e K，新 They are respectively: ； ； e A，新 and e K，新 Normalized and They are respectively: ； ； For matrix e A，新 The z-th element e A，z After normalization, we get The corresponding element The formula is: (2)； In equation (2), Representation matrix The z-th element; e A,z Represents matrix e A，新 The z-th element; Represents matrix e A，新 The mean of all elements in the dataset; Represents matrix e A，新 The standard deviation of all elements in the sample; Matrix e K，新 The x-th element e K，x After normalization, we obtain the matrix. The corresponding element The formula is: (3)； In equation (3), Representation matrix The x-th element; e K,x Represents matrix e K，新 The x-th element; Represents matrix e K，新 The mean of all elements in the dataset; Represents matrix e K，新 The standard deviation of all elements in the matrix.

3. The cold start optimization method for long sequence recommendation systems based on Transformer according to claim 1, characterized in that, The formula for the cross-entropy loss function in step S3 is: (6)； In equation (6), y i In real-world scenarios, 1 represents whether a user interacts with an item in the i-th multi-source heterogeneous data set; 0 represents the number of multi-source heterogeneous data sets if there is no interaction.

4. The cold start optimization method for long sequence recommendation systems based on Transformer according to claim 1, characterized in that, In step S3, The update formulas for the weights of the hidden layer and the output layer are as follows: (7)； In equation (7), This represents the weight matrix of the hidden layer or the output layer at the (t+1)th iteration. This represents the weight matrix of the hidden layer or the output layer at the t-th iteration; Indicates the learning rate; This is the gradient of the loss function with respect to the weights of the hidden or output layer; the weight matrices of the hidden and output layers are continuously iterated during training.

5. The cold start optimization method for long sequence recommendation systems based on Transformer according to claim 3, characterized in that, When L is less than 0.05, it indicates that the training of the meta-learning model of the multilayer perceptron structure is complete.

Images