An article recommendation method based on large model and contrastive learning

Abstract

The present invention provides an article recommendation method based on a big model and contrastive learning, comprising the following steps: S1: counting a list of articles clicked by users; S2: generating a list of article summaries; S3: constructing a model input data pair; S4: dividing into batches, performing preprocessing and input preparation; S5: generating an optimal model; S6: recommending articles through the optimal model; the present invention uses the most intelligent big model system currently available, and due to the emergence effect of the big model, the intention understanding and intelligence of the big model are greatly improved compared with traditional methods; using a contrastive learning algorithm, combined with the big model to make recommendations to users, the results are more in line with user ideas, increase user stickiness, and improve user satisfaction.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an article recommendation method based on large models and contrastive learning. Background Art

[0002] A large model refers to a machine learning model with a large parameter scale and complex computational structure. It is usually composed of a deep neural network with billions or even hundreds of billions of parameters. It is designed to improve the model's expressiveness and predictive performance and to be able to handle complex tasks and data. Large models learn complex patterns and features by training on massive amounts of data. They have stronger generalization capabilities and can make accurate predictions on unseen data. When the training data and parameters of a machine learning model continue to expand until they reach a certain critical scale, its ability to automatically learn from the original training data and discover new, higher-level features and patterns is called "emergence capability". Machine learning models with emergent capabilities are considered to be large models in an independent sense, which is the biggest difference between large models and small models.

[0003] Contrastive learning is a machine learning method that aims to learn useful representations of data by comparing similar and dissimilar data samples. The main purpose is to enable the model to learn to map similar samples to similar representations to capture the basic structure and relationship between different data points. It is usually achieved by training algorithms to predict whether two data are from the same category. Contrastive learning has been applied in various businesses, such as image recognition, natural language processing, and speech recognition.

[0004] In forums, news websites, Weibo, book bars and other similar application environments that focus on reading articles, valuable articles are provided for users to read. In earlier designs, applications generally adopt a classified reading method, and users search for articles they want to read by selecting their favorite categories. In recent years, with the development of machine learning, general applications will actively recommend articles to users based on their preferences. These user preferences often come from their reading lists. Therefore, how to recommend the articles that users want to read most based on their usage habits is a core business of article reading applications. In existing recommendation algorithms, because they use traditional machine learning methods, they are not intelligent enough and do not understand users' intentions deeply enough. The recommended articles are often not enough to meet users' needs. In the long run, this will lead to a decrease in user stickiness to the platform, a loss of user activity on the platform, and a decline in business. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an article recommendation method based on large model and contrastive learning.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An article recommendation method based on a large model and contrastive learning includes the following steps:

[0008] S1: Count the user's clicked article list;

[0009] Count the clicked article lists of all users in the current reading application system. Let the number of all users be n, then the clicked article list of n users is recorded as ;

[0010] Specifically, the clicked article list of the i-th user , among which, there are j articles in total, Indicates the jth article, click on the article list The order of articles in is the chronological order in which users click on the articles.

[0011] S2: Generate a list of article abstracts;

[0012] Get all the articles in the clicked article list, set the word count interval U of the article summary in the big model, and the big model traverses each article, generates an article summary within U for each article, and stores it in a list, which is recorded as the article summary list , among which, there are j article abstracts, Represents the jth article abstract. The order of article abstracts in the article abstract list N corresponds to the order in the clicked article list.

[0013] S3: construct model input data pairs;

[0014] Generate model input data pairs based on the article abstract list corresponding to each user;

[0015] Each user clicks on j articles in total, corresponding to j article summaries. The corresponding model input data pairs are constructed based on the user's click history;

[0016] Specifically, the generated model input data pairs are: , , ... , one user builds j-2 model input data pairs, then n users generate n*(j-2) model input data pairs;

[0017] Furthermore, the first part of the model input data pair, namely the summary of the user's historical clicked articles, is the input of the user tower, specifically ; The second part of the model input data pair is the user's next clicked article summary, which is the input of the article tower. ; Among them, the article abstract is a text sequence consisting of words and subwords.

[0018] S4: batch division, preprocessing and input preparation;

[0019] The following sub-steps are included:

[0020] S41: Divide training data batches;

[0021] The training data is a model input data pair, and the model input data pair is divided into p batches according to actual needs, each batch contains W model input data pairs; W is a power of 2;

[0022] Furthermore, the model input data pairs of the same user are divided into p batches respectively;

[0023] S42: Data preprocessing and input preparation;

[0024] The following sub-steps are included:

[0025] S421: preprocessing the model input data pair;

[0026] Preprocess the model input data pairs in batches, use the large model to convert the user's historical clicked article summaries in step S3 into embedding sequences; use the Tokenizer tool to generate token sequences from the user's next clicked article summaries in step S3;

[0027] The large models include deep learning models such as GPT, BERT, T5, and RoBERTa;

[0028] Each word or subword in the summary of the article clicked by the user is encoded by the large model to obtain a set of embedded vectors. Each word or subword corresponds to a vector, and the sequence formed by these vectors is the embedding sequence;

[0029] The Tokenizer tool includes a BERT tokenizer or other tokenizer tools;

[0030] The token sequence refers to a set of integer index sequences converted by the Tokenizer tool into the next article summary clicked by the user;

[0031] S422: preparing the preprocessed sequence for input;

[0032] Transform the embedding sequence and token sequence through the mapping layer transformation so that the embedding sequence and token sequence have the same dimension;

[0033] The mapping layer includes a fully connected layer, a linear transformation, etc.

[0034] S5: Generate the optimal model;

[0035] The following sub-steps are included:

[0036] S51: Comparative learning of users and articles;

[0037] The embedding sequence in step S4 is used as the input of the user Transformer model, and the output is the user embedding;

[0038] The token sequence in step S4 is used as the input of the article Transformer model, and the output is the article embedding;

[0039] The user Transformer model and article Transformer model are two independent Transformer architecture models;

[0040] The contrast loss is calculated by the following formula:

[0041] ;

[0042] Among them, loss is the contrast loss, The user embedding for the i-th user in a batch, is the article embedding of the i-th user in a batch, the s function is the distance calculation function of two vectors, is the article embedding of the jth user in the first batch, and e is the natural logarithm;

[0043] S52: continuously update the parameters in the model to generate the optimal model;

[0044] Repeat steps S4-S51 to iterate and continuously update the parameters of the neural network in the user Transformer model and the article Transformer model to generate the optimal model.

[0045] S6: Recommend articles through the optimal model;

[0046] List of articles clicked by the i-th user , a total of j articles were clicked in chronological order;

[0047] Statistics of all articles currently read in the application system , a total of k articles;

[0048] by As the input of the article tower, get the article embedding, , a total of k; As the input of the user tower, the user embedding of user i is obtained, that is, ;

[0049] The embedding vector list of j articles in the article click list excluding the i-th user ; Do vector distance calculation to get:

[0050] ;

[0051] There are kj calculation results in total, and the calculation results are sorted from near to far according to the distance, which is the order of recommended articles to the i-th user.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The article recommendation method based on big model and contrastive learning proposed in the present invention uses the big model to generate article abstracts, and uses the embedding generated by the big model as the input of the user tower Transformer neural network; generates multiple data pairs of historical records and next article clicks when the user browses the article list, and uses the token sequence converted from this data pair as the input of the model; uses the contrastive learning method to train the two Transformer model neural networks of the user tower and the article tower; uses the emergence ability of the big model to process data, combines the contrastive learning method, and uses a dual-tower structure to train two Transformer model neural networks; combines the ability of the big model with the deep neural network in the article recommendation business to obtain better results;

[0054] The present invention uses the most intelligent big model system currently available. Due to the emergence effect of the big model, the intention understanding and intelligence of the big model have been greatly improved compared with traditional methods. The contrastive learning algorithm is used in combination with the big model to make recommendations to users, and the results are more in line with user ideas, increasing user stickiness and improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of the steps of an article recommendation method based on a large model and contrastive learning according to the present invention;

[0056] Figure 2 This is a schematic diagram of using a large model to generate summaries and embedding functions in an article recommendation method based on a large model and contrastive learning in the present invention. DETAILED DESCRIPTION

[0057] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0058] like Figure 1 , Figure 2As shown, an article recommendation method based on a large model and contrastive learning includes the following steps:

[0059] S1: Count the user's clicked article list;

[0060] Count the clicked article lists of all users in the current reading application system. Let the number of all users be n, then the clicked article list of n users is recorded as ;

[0061] Specifically, the clicked article list of the i-th user , among which, there are j articles in total, Indicates the jth article, click on the article list The order of articles in is the chronological order in which users click on the articles.

[0062] S2: Generate a list of article abstracts;

[0063] Get all the articles in the clicked article list, set the word count interval U of the article summary in the big model, and the big model traverses each article, generates an article summary within U for each article, and stores it in a list, which is recorded as the article summary list , among which, there are j article abstracts, Represents the jth article abstract. The order of article abstracts in the article abstract list N corresponds to the order in the clicked article list.

[0064] Since the length of articles may vary greatly, directly using them as model input may cause data skew; therefore, we first use the article summary generation capability of the large model to generate article summaries of similar lengths for articles of different lengths.

[0065] S3: construct model input data pairs;

[0066] Generate model input data pairs based on the article abstract list corresponding to each user;

[0067] Each user clicks on j articles in total, corresponding to j article summaries. The corresponding model input data pairs are constructed based on the user's click history;

[0068] Specifically, the generated model input data pairs are: , , ... , one user builds j-2 model input data pairs, then n users generate n*(j-2) model input data pairs;

[0069] Furthermore, the first part of the model input data pair, namely the summary of the user's historical clicked articles, is the input of the user tower, specifically ; The second part of the model input data pair is the user's next clicked article summary, which is the input of the article tower. ; Among them, the article abstract is a text sequence consisting of words and subwords.

[0070] In order to effectively predict the next article, it is usually necessary to rely on at least two previous articles; if only the first article is used to predict the second article, it may not fully capture the user's preference patterns.

[0071] S4: batch division, preprocessing and input preparation;

[0072] The following sub-steps are included:

[0073] S41: Divide training data batches;

[0074] The training data is a model input data pair, and the model input data pair is divided into p batches according to actual needs, each batch contains W model input data pairs; W is a power of 2;

[0075] Furthermore, the model input data pairs of the same user are divided into p batches respectively;

[0076] By dividing the model input data pairs of the same user into p batches, the possibility of the model input data pairs of the same user being assigned to the same batch is avoided, preventing the model from seeing multiple click histories of the same user in the same batch, thus avoiding data leakage.

[0077] S42: Data preprocessing and input preparation;

[0078] The following sub-steps are included:

[0079] S421: preprocessing the model input data pair;

[0080] Preprocess the model input data pairs in batches, use the large model to convert the user's historical clicked article summaries in step S3 into embedding sequences; use the Tokenizer tool to generate token sequences from the user's next clicked article summaries in step S3;

[0081] The large models include deep learning models such as GPT, BERT, T5, and RoBERTa;

[0082] Each word or subword in the summary of the article clicked by the user is encoded by the large model to obtain a set of embedded vectors. Each word or subword corresponds to a vector, and the sequence formed by these vectors is the embedding sequence;

[0083] The Tokenizer tool includes a BERT tokenizer or other tokenizer tools;

[0084] The token sequence refers to a set of integer index sequences converted by the Tokenizer tool into the next article summary clicked by the user;

[0085] Generate a token sequence from the article summary through Tokenizer; Converting text into a token sequence is a common encoding form. Converting natural language into a token sequence through encoding. Usually, the first step of NLP processing is to tokenize natural language. Common methods include Wordpiece and BPE. Large models also have their own Tokenizer. This invention does not specify a specific large model or a specific tokenization method.

[0086] Generating embedding vectors is a basic function of the big model. The big model generates embedding vectors that represent the deep representation and features of the article from the article summary. This embedding vector, enabled by the big model, already has the deep representation of the article and can represent the article itself in the vector dimension.

[0087] Convert the articles that users have browsed in the past into embeddings in order. Multiple articles can generate a set of ordered embedding vectors through a large model, and each embedding vector represents an article.

[0088] S422: preparing the preprocessed sequence for input;

[0089] Transform the embedding sequence and token sequence through the mapping layer transformation so that the embedding sequence and token sequence have the same dimension;

[0090] The mapping layer includes a fully connected layer, a linear transformation, etc.;

[0091] Convert the dimensions of the embedding sequence and the token sequence to the same size, so that each sequence input to the Transformer model is consistent in dimension, thus meeting the model's requirements for the input format.

[0092] The generated embedding vector sequence is used as the input of the next model. This embedding vector sequence is also a set of token sequences. This model uses the same Transformer architecture as the large model. It inputs a set of token sequences and can output an embedding vector containing representations and features. Here, the embedding vector sequence of the article list generated by the large model in the previous process is used as the input token sequence of the Transformer model, and the final output is a user embedding vector.

[0093] S5: Generate the optimal model;

[0094] The following sub-steps are included:

[0095] S51: Comparative learning of users and articles;

[0096] The embedding sequence in step S4 is used as the input of the user Transformer model, and the output is the user embedding;

[0097] The token sequence in step S4 is used as the input of the article Transformer model, and the output is the article embedding;

[0098] The user Transformer model and article Transformer model are two independent Transformer architecture models;

[0099] The contrast loss is calculated by the following formula:

[0100] ;

[0101] Among them, loss is the contrast loss, The user embedding for the i-th user in a batch, is the article embedding of the i-th user in a batch, the s function is the distance calculation function of two vectors, is the article embedding of the jth user in the first batch, and e is the natural logarithm;

[0102] S52: continuously update the parameters in the model to generate the optimal model;

[0103] Repeat steps S4-S51 to iterate and continuously update the parameters of the neural network in the user Transformer model and the article Transformer model to generate the optimal model;

[0104] A dual-tower structure is used to train the entire network using contrastive learning. The input of the user tower is the user's article browsing list, and the output is the user embedding vector. The input of the article tower is the next article browsed by the user, and the output is the article embedding vector. The contrastive learning method is used to calculate the vector distance between the user embedding vector and the article embedding vector, so that articles related to the user are closer to the user, and irrelevant articles are farther away from the user. This method is used to train the entire network.

[0105] S6: Recommend articles through the optimal model;

[0106] List of articles clicked by the i-th user , a total of j articles were clicked in chronological order;

[0107] Statistics of all articles currently read in the application system , a total of k articles;

[0108] by As the input of the article tower, get the article embedding, , a total of k; As the input of the user tower, the user embedding of user i is obtained, that is, ;

[0109] The embedding vector list of j articles in the article click list excluding the i-th user ; Do vector distance calculation to get:

[0110] ;

[0111] There are kj calculation results in total, and the calculation results are sorted from near to far according to the distance, which is the order of recommended articles to the i-th user.

[0112] For the pre-trained dual-tower model, the user tower inputs the user's browsed article list data, and after passing through the large model and the user Transformer model, the final user embedding represents the representation and characteristics of the user's past article reading habits; because both the large model and the user Transformer model have self-attention mechanisms, the user embedding not only contains the article content representation, but also contains the hidden representation of the order in which the user browses the articles, representing the user's reading habits and preferences, as well as the changing patterns of these habits and preferences. The article tower inputs any article in the system, and through the article Transformer model, the article embedding obtained contains the representation and characteristics of the article; the user embedding and the embedding of all articles in the system are vectorized, and the distance between the vectors is used as the basis for recommending articles to the user. The closer the distance, the more it meets the user's current reading needs.

[0113] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. An article recommendation method based on large model and contrastive learning, characterized by: The following steps are involved: S1: Count the user's clicked article list; Count the clicked article lists of all users in the current reading application system, and record the number of all users as n. Then the clicked article lists of n users are recorded as M = {M1, M2, M3, ...M n }; Specifically, the clicked article list M of the i-th user i ={A1, A2, A3, ... A j }, among which there are j articles, A j Indicates the jth article, click on the article list M i The order of articles in the table is the time sequence in which users click on the articles; S2: Generate a list of article abstracts; Get all the articles in the clicked article list, generate article summaries for each article through the big model, and store them in the list; Specifically, all articles in the clicked article list are obtained, and the word count interval U of the article summary is set in the big model. The big model traverses each article, generates an article summary within U for each article, and stores it in a list, which is recorded as the article summary list N = {a1, a2, a3, ...a j }, where there are j article abstracts, a j represents the jth article abstract, and the order of article abstracts in the article abstract list N corresponds to the order in the clicked article list; S3: construct model input data pairs; Generate model input data pairs based on the article abstract list corresponding to each user; Each user clicks on j articles in total, corresponding to j article summaries. The corresponding model input data pairs are constructed based on the user's click history; The generated model input data pairs are: [{a1, a2}, {a3}], [{a1, a2, a3}, {a4}], [{a1, a2, a3, a4}, {a5}]...[{a1, a2, a3,...a j-1 }, {a j }], one user builds j-2 model input data pairs, then n users generate n*(j-2) model input data pairs; Furthermore, the first part of the model input data pair, i.e., the summary of the user's historical clicked articles, is the input of the user tower, specifically [{a1, a2}, {a1, a2, a3}, {a1, a2, a3, a4}, ..., {a1, a2, a3, ...a j-1 }]; the latter part of the model input data pair is the summary of the next article clicked by the user, which is the input of the article tower, specifically [{a3}, {a4}, {a5}, ..., {a j }]; where the article abstract is a text sequence consisting of words and subwords; S4: batch division, preprocessing and input preparation; Divide the model input data pairs into batches according to actual needs, preprocess the model input data pairs by batches, and generate embedding sequences and token sequences; S5: Generate the optimal model; The following sub-steps are included: S51: Comparative learning of users and articles; The embedding sequence in step S4 is used as the input of the user Transformer model, and the output is the user embedding; The token sequence in step S4 is used as the input of the article Transformer model, and the output is the article embedding; The user Transformer model and article Transformer model are two independent Transformer architecture models; Calculate contrast loss; S52: continuously update the parameters in the model to generate the optimal model; Repeat steps S4-S51 to iterate and continuously update the parameters of the neural network in the user Transformer model and the article Transformer model to generate the optimal model; S6: Recommend articles using the optimal model.

2. The article recommendation method based on large model and contrastive learning as claimed in claim 1, characterized in that: Step S4 includes the following sub-steps: S41: Divide training data batches; The training data is a model input data pair, and the model input data pair is divided into p batches according to actual needs, each batch contains W model input data pairs; W is a power of 2; Furthermore, the model input data pairs of the same user are divided into p batches respectively; S42: Data preprocessing and input preparation; The following sub-steps are included: S421: preprocessing the model input data pair; Preprocess the model input data pairs in batches, use the large model to convert the user's historical clicked article summaries in step S3 into embedding sequences; use the Tokenizer tool to generate token sequences from the user's next clicked article summaries in step S3; Each word or subword in the summary of the article clicked by the user is encoded by the large model to obtain a set of embedded vectors. Each word or subword corresponds to a vector, and the sequence formed by these vectors is the embedding sequence; The Tokenizer tool includes a BERT tokenizer or other tokenizer tools; The token sequence refers to a set of integer index sequences converted by the Tokenizer tool into the next article summary clicked by the user; S422: preparing the preprocessed sequence for input; The embedding sequence and token sequence are transformed through the mapping layer transformation so that the embedding sequence and token sequence have the same dimension.

3. The article recommendation method based on large model and contrastive learning as claimed in claim 1, characterized in that: In step S5, the contrast loss is calculated by the following formula: Among them, loss is the contrast loss, The user embedding for the i-th user in a batch, is the article embedding of the i-th user in a batch, the s function is the distance calculation function of two vectors, is the article embedding of the jth user in the first batch, and e is the natural logarithm; Each batch contains W model input data pairs; W is the number of model input data pairs, which is a power of 2.

4. The article recommendation method based on large model and contrastive learning as claimed in claim 3, characterized in that: The specific contents of step S6 are as follows: List of articles clicked by the i-th user M i ={A1, A2, A3, ... A j }, clicked j articles in chronological order; Count all articles in the current reading application system B = {B1, B2, B3, ...B j }, total k articles; With B = {B1, B2, B3, ... B j } as the input of the article tower, get the article embedding, There are k in total; M i ={A1, A2, A3, ... A j } as the input of the user tower, and get the user embedding of user i, that is, The embedding vector list of j articles in the article click list excluding the i-th user Doing vector distance calculation yields: There are kj calculation results in total, and the calculation results are sorted from near to far according to the distance, which is the order of recommended articles to the i-th user.

Images