Recommendation methods that enhance user behavior features using large language models

Abstract

A recommendation method that enhances user behavior features using a large language model generates a semantic representation embedding vector for each item based on its name, category, and description, thus creating a semantic vector library for the items. A training sample library is constructed based on the dataset records. Each training sample includes user features, target item features, user's historical item interaction sequence features, and a label indicating whether there was an interaction. After constructing and training the recommendation model using the semantic vector library and the sample training library with a two-layer filtering approach, real-time recommendations are made online. This invention generates semantic representation embedding vectors for items by collecting their titles, descriptions, and reviews. By filtering and enhancing the sequence features of items based on the similarity between semantic vectors, the accuracy of the recommendation model's predictions is improved.

Description

Technical Field

[0001] This invention relates to a technology in the field of information recommendation, specifically a recommendation method that enhances user behavior features using a large language model. Background Technology

[0002] Recommendation technology is crucial in the era of information overload, but the current paradigm that relies on behavioral sequences struggles to fully utilize the world knowledge in item texts, and semantic and intent capture is limited; at the same time, industrial scenarios have high requirements for latency and computing power, which challenges the powerful understanding capabilities of fusion large models. Summary of the Invention

[0003] This invention addresses the issues of high optimization costs and large data transmission volumes in existing technologies, which can easily lead to online service latency. It proposes a recommendation method that enhances user behavior features using a large language model. By collecting item titles, descriptions, reviews, etc., it generates semantic representation embedding vectors for items. Then, by filtering and enhancing the sequence features of items from the user's historical interactions through the similarity between semantic vectors, it improves the prediction accuracy of the recommendation model.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a recommendation method that enhances user behavior features using a large language model, comprising:

[0006] Step 1: Generate a semantic representation embedding vector for each item based on its name, category, and description, thereby generating a semantic vector library for the items;

[0007] Step 2: Based on the records in the dataset, construct a training sample library. Each training sample includes user features, target item features, user item historical interaction sequence features, and a label indicating whether there was an interaction.

[0008] Step 3: After constructing and training the recommendation model using the semantic vector library obtained in Step 1 and the sample training library obtained in Step 2 with a two-layer filtering method, real-time recommendations are made through the trained recommendation model in the online stage.

[0009] The recommendation model includes: a feature embedding unit, a semantic query unit, a semantic similarity calculation unit, a semantic enhancement unit, and a feature cross prediction unit. Specifically: the feature embedding unit reads training samples from the sample training library and converts all features into embedding vectors; the semantic query unit searches for the corresponding semantic representation embedding vector from the semantic vector library based on the IDs of the target item and each item in the historical interaction sequence; the semantic similarity calculation unit calculates the semantic similarity between the target item and each item in the historical interaction sequence based on the semantic representation embedding vectors obtained from the semantic query unit, obtaining the weights of each item in the sequence; the semantic enhancement unit adjusts the embedding vectors of the historical interaction sequence obtained from the feature embedding unit based on the weights obtained from the semantic similarity calculation unit; and the feature cross prediction unit uses the historical interaction sequence embedding vectors adjusted by the semantic enhancement unit, along with the remaining embedding vectors output by the feature embedding unit, to perform feature cross-prediction and predict the probability of user interaction with the target item.

[0010] The two-layer filtering method includes: in the coarse-grained filtering stage, the embedding vector of the historical interaction sequence of the sample is calculated by the feature embedding unit, and the semantic representation embedding vector and the semantic similarity between items are obtained by the semantic query unit and the semantic similarity calculation unit respectively; in the fine-grained filtering stage, the embedding vector of the historical interaction sequence is fine-tuned by the semantic enhancement unit according to the semantic similarity between items, and the prediction result is obtained by the feature cross prediction unit.

[0011] Technical effect

[0012] This invention, under the constraints of low latency and limited cost in industrial applications, incorporates a two-layer filtering approach to enhance the powerful semantic understanding capabilities of large language models, thereby improving recommendation prediction capabilities. Compared with existing technologies, this invention can significantly improve the area under the receiver operating characteristic curve (AUC) and the AUC (gAUC) by user group, without disrupting existing recommendation chains. It is compatible with any recommendation model already in production, significantly improving the return on investment. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention;

[0014] Figure 2 This is a schematic diagram of the recommended model for an example. Detailed Implementation

[0015] like Figure 1 As shown in this embodiment, a recommendation method for enhancing user behavior features using a large language model includes:

[0016] Step 1: Semantic Representation Embedding Vector Generation Stage: Based on the item's name, category, and description, a sentence representing the item is constructed and input into the large model to generate the corresponding semantic representation embedding vector. This specifically includes:

[0017] 1-1 item catalog Each item in Extract all its meta-text information and concatenate it into a text representation of the item according to the template "field name: field content". .

[0018] The metatext information includes: original text such as name, category, and description.

[0019] For example, the text representation For: Name: Model name. Category: Digital product. Description: The latest generation of mobile phones released by a certain company, using the screen of a certain model throughout.

[0020] 1-2. Represent the text Input into a large language model to generate corresponding text representation embedding vectors. Dimension of a vector The dimensions pre-defined for large language models include: This indicates the large language model used.

[0021] The large language model mentioned above uses, but is not limited to, GPT-4o, Qwen2.5 or their corresponding embedding models, such as text-embedding-ada-002, bge-m3, etc., which can directly output text representation embedding vectors.

[0022] Step 2, Semantic Representation Embedding Vector Storage Stage: Store the item ID and the corresponding semantic representation embedding vector obtained in Step 1-2 according to key-value pairs. The semantic vectors of the items are stored in the form of the item's semantic vector library. .

[0023] Step 3, Semantic Representation Embedded Vector Update Stage: The information update trigger module monitors changes in item information and adds, updates, or deletes records in the vector table based on these changes.

[0024] Step 4: Building the training sample library: Based on the metadata in the dataset, user features, item features and labels are spliced ​​together to form training samples and compose the sample library.

[0025] The user features include: profile features, context features, and user history interaction sequence features.

[0026] The profile features include: user ID, gender, and age.

[0027] The contextual features include: interaction time and page.

[0028] The aforementioned user history interaction sequence features include: item IDs that the user interacted with within a certain period of time, arranged in chronological order of interaction time, represented as follows: ,in Indicates user For a period of time Each interactive item ID, , is the length of the sequence.

[0029] The characteristics of the items include: attribute characteristics and statistical characteristics.

[0030] The attribute features include: item ID and item category.

[0031] The statistical features mentioned include: number of exposures and number of clicks.

[0032] The labels indicate whether the user interacts with the target item, serving as a supervisory signal for model training.

[0033] Step 5: Construct and train the recommendation model using a two-layer filtering approach. In the coarse-grained filtering stage, the feature embedding unit calculates the embedding vector of the historical interaction sequence of the samples. The semantic query unit and semantic similarity calculation unit then sequentially obtain the semantic representation embedding vector and the semantic similarity between items, respectively. In the fine-grained filtering stage, the semantic enhancement unit fine-tunes the embedding vector of the historical interaction sequence based on the semantic similarity between items, and the feature cross-prediction unit obtains the prediction result. Specifically, this includes:

[0034] 5-1 The feature embedding unit calculates the embedding vector of the historical interaction sequence of the sample, specifically including:

[0035] i uses one-hot encoding to transform the discrete features of the sample into vector form, and calculates the corresponding embedding vector based on the trainable embedding table in the layer.

[0036] ii. Based on the interval where the dense feature is located, divide the interval into a finite number of categories and calculate the embedding vector corresponding to each category;

[0037] iii Based on historical interaction sequences Each item ID in The corresponding embedding vector is obtained according to step i. Then, they form an embedded vector sequence. As an embedding vector of the historical interaction sequence, where: .

[0038] The discrete features mentioned include, but are not limited to, item ID, category, etc.

[0039] The dense features mentioned include, but are not limited to, the number of interactions and the probability of interaction.

[0040] 5-2 Semantic query unit based on user's historical interaction sequence characteristics and target items and from the semantic vector library Query its corresponding semantic representation embedding vector ,in: .

[0041] 5-3 The semantic similarity calculation unit uses semantic representation embedding vectors to calculate the items in each sequence. and target items semantic similarity ,in: For semantic similarity function, .

[0042] The semantic similarity function One of the following can be used, but is not limited to:

[0043] a) Cosine similarity function: Calculate the cosine value of the embedding vectors of both.

[0044] b) Distance function: or ,pass Distance measures the similarity between embedded vectors.

[0045] c) Attention function of Deep Interest Network (DIN) Attention between embedding vectors is learned using activation units in the deep interest network proposed by Alibaba Group. This type of attention is widely used in sequence recommendation models.

[0046] d) Cross-attention function Similarity is learned using a cross-attention mechanism, the structure of which is computed using the cross-attention unit in the Transformer model.

[0047] The 5-4 semantic enhancement unit enhances the embedding vector of the historical interaction sequence obtained in step 5-1 based on the semantic similarity obtained in step 5-3, specifically by using any of the following methods:

[0048] a) Directly use semantic similarity as a weight to perform a weighted summation of the embedding vectors of historical interaction sequence features, and construct a new embedding vector for that feature. ,in: , .

[0049] b) Using semantic similarity as a reference value for sequence truncation, filter out items with low relevance to the target item from the user's historical interaction sequence to obtain a new embedding vector. ,in: , Sequence mask , This is the preset sequence filtering threshold.

[0050] The 5-5 feature cross prediction unit predicts based on the enhanced embedding vector. The embedding vectors corresponding to the discrete features mentioned in step 5-1i and the embedding vectors corresponding to the dense features mentioned in step 5-1ii are used to generate the model prediction probability of user-target item interaction.

[0051] The feature cross prediction unit is implemented using, but is not limited to, any of the following techniques:

[0052] a) Wide & Deep model: The deep module performs deep cross-interaction on the features that need to be crossed, while the wide module directly inputs the important features into the final prediction layer;

[0053] b) PNN model: Introduces a product module to cross features;

[0054] c) DCN model: Features are crossed using both deep modules and cross modules;

[0055] d) DIN model: Use activation units to learn interest signals from user history and amplify those signals to target items.

[0056] Through specific practical experiments, the training batch size was set to 1024 on an NVIDIA GeForce RTX4090D hardware environment, employing a training early stopping strategy with a tolerance of 2 rounds and a learning rate of 0.005. Experiments were conducted on four public datasets: Amazon Electronics, Amazon Books, KKBox, and Yelp. In a specific industrial scenario, we collaborated with a technology company to verify the effects on their private dataset. The experiments primarily verified the effectiveness of all similarity functions mentioned in steps 5-3 and all recommendation model structures mentioned in step 6-2, listing the results of the experimental groups with the greatest improvement. All experimental data are shown in Table 1, where the baseline represents the effect of the recommendation model without using this invention.

[0057] Table 1

[0058]

[0059] Compared with existing technologies, in experiments on multiple public datasets, this invention can achieve a relative improvement of up to 3.93% in AUC and 5.53% in gAUC compared to not using the framework; while in actual advertising recommendation scenarios in the industry, this invention achieved a relative improvement of 0.28% in AUC and 0.96% in gAUC.

[0060] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A recommendation method that enhances user behavior features using a large language model, characterized in that, include: Step 1: Generate a semantic representation embedding vector for each item based on its name, category, and description, thereby generating a semantic vector library for the items; Step 2: Based on the records in the dataset, construct a training sample library. Each training sample includes user features, target item features, user item historical interaction sequence features, and a label indicating whether there was an interaction. Step 3: After constructing and training the recommendation model using the semantic vector library obtained in Step 1 and the sample training library obtained in Step 2 with a two-layer filtering method, real-time recommendations are made through the trained recommendation model in the online stage. The recommendation model includes: a feature embedding unit, a semantic query unit, a semantic similarity calculation unit, a semantic enhancement unit, and a feature cross prediction unit. Specifically: the feature embedding unit reads training samples from the sample training library and converts all features into embedding vectors; the semantic query unit searches for the corresponding semantic representation embedding vector from the semantic vector library based on the IDs of the target item and each item in the historical interaction sequence; the semantic similarity calculation unit calculates the semantic similarity between the target item and each item in the historical interaction sequence based on the semantic representation embedding vectors obtained from the semantic query unit, obtaining the weights of each item in the sequence; the semantic enhancement unit adjusts the embedding vectors of the historical interaction sequence obtained from the feature embedding unit based on the weights obtained from the semantic similarity calculation unit; and the feature cross prediction unit uses the historical interaction sequence embedding vectors adjusted by the semantic enhancement unit, along with the remaining embedding vectors output by the feature embedding unit, to perform feature cross-prediction and predict the probability of user interaction with the target item. Step 3 specifically includes: 5-1 The feature embedding unit calculates the embedding vector of the historical interaction sequence of the sample, specifically including: i uses one-hot encoding to transform the discrete features of the sample into vector form, and calculates the corresponding embedding vector based on the trainable embedding table in the layer. ii. Based on the interval where the dense feature is located, divide the interval into a finite number of categories and calculate the embedding vector corresponding to each category; iii Based on historical interaction sequences Each item ID in The corresponding embedding vector is obtained according to step i. Then, they form an embedded vector sequence. As an embedding vector of the historical interaction sequence, where: ; 5-2 Semantic query unit based on user's historical interaction sequence characteristics and target items and from the semantic vector library Query its corresponding semantic representation embedding vector ,in: ; 5-3 The semantic similarity calculation unit uses semantic representation embedding vectors to calculate the items in each sequence. and target items semantic similarity ,in: For semantic similarity function, ; The semantic enhancement unit 5-4 enhances the embedding vector of the historical interaction sequence obtained in step 5-1 based on the semantic similarity obtained in step 5-3. The 5-5 feature cross prediction unit predicts based on the enhanced embedding vector. The embedding vectors corresponding to the discrete features mentioned in step 5-1i and the embedding vectors corresponding to the dense features mentioned in step 5-1ii are used to generate the model prediction probability of user-target item interaction.

2. The recommendation method for enhancing user behavior features using a large language model according to claim 1, characterized in that, The two-layer filtering method includes: in the coarse-grained filtering stage, the embedding vector of the historical interaction sequence of the sample is calculated by the feature embedding unit, and the semantic representation embedding vector and the semantic similarity between items are obtained by the semantic query unit and the semantic similarity calculation unit respectively; in the fine-grained filtering stage, the embedding vector of the historical interaction sequence is fine-tuned by the semantic enhancement unit according to the semantic similarity between items, and the prediction result is obtained by the feature cross prediction unit.

3. The recommendation method for enhancing user behavior features using a large language model according to claim 1 or 2, characterized in that, Step 1 specifically includes: 1-1 item catalog Each item in Extract all its meta-text information and concatenate it into a text representation of the item according to the template "field name: field content". ; 1-2. Represent the text Input into a large language model to generate corresponding text representation embedding vectors. Dimension of a vector The dimensions pre-defined for large language models include: This indicates the large language model used.

4. The recommendation method for enhancing user behavior features using a large language model according to claim 3, characterized in that, Step 2 specifically includes: 2-1 Semantic Representation Embedding Vector Storage Stage: Store the item ID and the corresponding semantic representation embedding vector obtained in step 1-2 according to key-value pairs. The semantic vectors of the items are stored in the form of the item's semantic vector library. ; 2-2. Semantic Representation Embedded Vector Update Stage: The information update trigger module monitors changes in item information and adds, updates, or deletes records in the vector table based on these changes; 2-3. Training Sample Library Construction Stage: Based on the metadata in the dataset, user features, item features, and labels are spliced ​​together to form training samples and compose a sample library; The user features include: profile features, context features, and user history interaction sequence features; The profile features include: user ID, gender, and age; The contextual features include: interaction time and page; The aforementioned user history interaction sequence features include: item IDs that the user interacted with within a certain period of time, arranged in chronological order of interaction time, represented as follows: ,in Indicates user For a period of time Each interactive item ID, , The length of the sequence; The item characteristics include: attribute characteristics and statistical characteristics; The attribute features include: item ID and item category; The statistical features mentioned include: number of impressions and number of clicks; The labels indicate whether the user interacts with the target item, serving as a supervisory signal for model training.

5. The recommendation method for enhancing user behavior features using a large language model according to claim 1, characterized in that, The semantic similarity function Implement using any of the following methods: a) Cosine similarity function: Calculate the cosine value of the embedding vectors of both vectors; b) Distance function: or ,pass Distance measures the similarity between embedded vectors; c) Attention function of Deep Interest Network (DIN) Attention between embedding vectors is learned using activation units in the deep interest network proposed by Alibaba Group. This type of attention is widely used in sequence recommendation models. d) Cross-attention function Similarity is learned using a cross-attention mechanism, the structure of which is computed using the cross-attention unit in the Transformer model.

6. The recommendation method for enhancing user behavior features using a large language model according to claim 1, characterized in that, Step 5-4, as described above, specifically involves: a) Directly use semantic similarity as a weight to perform a weighted summation of the embedding vectors of historical interaction sequence features, and construct a new embedding vector for that feature. ,in: , Or b) Using semantic similarity as a reference value for sequence truncation, filter out items with low relevance to the target item from the user's historical interaction sequence to obtain a new embedding vector. ,in: , Sequence mask , The length of the filtered sequence.

Images