A Serialization Recommendation Method Based on a Large Language Model

Abstract

This invention discloses a serialization recommendation method based on a large language model (LLM), comprising the following steps: A1. Input: Input a user set and an item set into the large language model (LLM). For each user, their historical interaction sequence, where each is an item that has been interacted with. Contextual information of the items. A2. Data Processing: a) Perform semantic clustering based on the logit representation of candidate items output by the LLM; b) Estimate semantic uncertainty based on the semantic clustering results; c) Adjust the decoding steps of item rating and / or sampling strategies according to the semantic uncertainty, and select the top-performing items as the final recommendation result; A3. Output a list of recommended items. This invention solves the problem of mismatch between the decoding strategy and recommendation objective in existing large language models for serialization recommendation; its core idea is to combine logit-based clustering and adaptive rating to improve the prediction of the next item.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and recommender systems, and more specifically, relates to a sequential recommendation method based on a large language model. Background Technology

[0002] Existing sequential recommendation techniques, particularly those based on Large Language Models (LLMs), have made significant progress. Leveraging their powerful natural language processing capabilities, LLMs can effectively handle multimodal data, including textual descriptions, visual features, and categorical attributes, thereby unifying different information sources and enabling flexible, high-quality recommendations based on complex user preferences. These models typically process user history and item metadata in an autoregressive manner to predict the next likely interactive item.

[0003] However, current serialization recommendation systems based on large language models mostly use decoding strategies designed for natural language processing tasks, such as beam search or greedy decoding, during the inference stage. This approach leads to the following problems and shortcomings: Goal mismatch: Natural language generation decoding strategies aim to generate fluent and coherent sequences of text, while the goal of recommendation systems is to accurately predict the next single item a user might be interested in. This difference in goals leads to the inefficiency or poor performance of existing decoding strategies in recommendation tasks. For example, bundle search may incur unnecessary computational overhead, while greedy decoding may miss high-probability candidate items.

[0004] The issue of semantic equivalence is overlooked: In recommendation scenarios, different items or sequences of items may satisfy the same underlying user needs, i.e., semantic equivalence exists. For example, if a user wants to buy wireless headphones, several similar products from different brands may be equally attractive. Standard decoding methods, such as bundle search, only focus on the probability at the token level, without considering these semantic relationships. When the probability quality is distributed across semantically equivalent items, uncertainty may be overestimated, thus affecting the accuracy of recommendations.

[0005] Lack of effective modeling and utilization of uncertainty: Although existing research has focused on uncertainty quantification (UQ) of large language models, such as entropy measurement methods based on word probability distribution, or allowing models to self-describe confidence, how to effectively utilize uncertainty to guide the decoding process in the recommendation decoding stage, especially in combination with semantic uncertainty, remains a challenge. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a serialization recommendation method based on a large-scale language model.

[0007] To achieve the aforementioned objectives, the technical solution adopted by this invention includes: a serialization recommendation method based on a large-scale language model, comprising the following steps: A serialization recommendation method based on a large language model includes the following steps: A1. Input: Input user set in a large language model (LLM) and item collection .

[0008] For each user Its historical interaction sequence Each of them It is an item that has already been interacted with.

[0009] Context information of item s For example, the item's title, category, brand, description, and other attributes; The serialized recommendation method based on large language models (LLM) significantly improves the intelligence level of the recommendation system by combining user historical interaction sequences and item context information.

[0010] A2. Data Processing: a) Perform semantic clustering based on the logit representation of candidate items output by a large language model; b) Estimating semantic uncertainty based on semantic clustering results; c) Adjust the decoding steps of item rating and / or sampling strategies based on semantic uncertainty, and select the top-performing items as the final recommendation results; A3. Output: Predicting users In its historical interaction sequence The next item most likely to be interacted with. ; Alternatively, it could be a sorted list of recommended items, representing the final item recommendations.

[0011] This invention proposes an uncertainty-aware semantic decoding framework (hereinafter referred to as USD) to address the mismatch between decoding strategies and recommendation goals in existing large-scale language models for serialized recommendation. Its core idea is to combine logit-based clustering and adaptive scoring to improve the prediction of the next item.

[0012] Further, step a) includes obtaining the logit vector of the candidate items. Calculate the similarity between logit vectors And based on similarity threshold Items are categorized into different semantic clusters.

[0013] Furthermore, the user's historical interaction sequence Contextual information of candidate items The information is input into a pre-trained LLM, which processes it in an autoregressive manner, and its prediction process is as follows: LLM generates the logit vector for a set of candidate items. ; Use the logit vector of candidate items and Calculate the pairwise similarity between them, such as cosine similarity: ; Based on the preset similarity threshold Items with similar logit vectors are grouped into the same semantic cluster, and the semantic equivalence relation is defined as follows: ,in It is an indicator function; it divides the candidate item set into several semantic clusters. .

[0014] Further, step b) includes calculating the aggregation probability of each semantic cluster. And calculate semantic entropy based on these cluster probabilities. Accuracy can be improved by optimizing smoothing strategies and real-time update mechanisms.

[0015] Furthermore, for each formed semantic cluster Its probability is calculated as the sum of the original probabilities of all items within the cluster: ,in It is an item provided by LLM. The probability of; Calculate the overall semantic entropy based on the probability distribution of all semantic clusters. Quantify the uncertainty of user preferences at the cluster level.

[0016] Furthermore, the steps for adaptively adjusting item ratings include: combining the item's original probability. The probability of its semantic cluster, the size of the cluster, and the calculated semantic entropy. To generate the final score .

[0017] Furthermore, adaptive scoring: for each candidate item Its final score ;in It is the base probability of an item. It is a balancing weight; function ,in It is an item The semantic cluster to which it belongs It is the size of the cluster. The degree to which entropy is controlled.

[0018] Furthermore, the steps for adaptively adjusting the sampling strategy include: based on the calculated semantic entropy... Adjusting the sampling temperature during the decoding process .

[0019] Furthermore, adaptive sampling temperature: the temperature used for item sampling during the decoding process. ;in It is the base temperature. The driving force of uncertainty on temperature is adjusted. Higher uncertainty leads to higher sampling temperatures, encouraging exploration; lower uncertainty causes sampling to focus more on high-probability items. This is based on the final adaptive score. Candidate items are sorted and combined with adaptive sampling temperature. The top-performing items will be selected as the final recommendations.

[0020] Compared with the prior art, the advantages of the present invention include: 1. This paper first points out the limitations of decoding strategies in existing LLM-based recommendation systems and introduces a new perspective on semantic uncertainty to understand recommendation tasks. It captures deep semantic reasoning information in LLM, solves the semantic misalignment problem between IDs and natural language descriptions in traditional recommendation, and significantly improves the equivalent recognition ability of synonymous products.

[0021] 2. Efficient Semantic Clustering Algorithm: A high-efficiency semantic clustering algorithm based on logit was developed, which captures the preference equivalence relationship between candidate items by analyzing the similarity of their probability distributions. This differs from traditional methods that rely solely on the item's own attributes for clustering.

[0022] 3. Uncertainty-guided scoring mechanism: An uncertainty-guided scoring mechanism is designed, which balances recommendation quality and computational efficiency by incorporating semantic information (semantic cluster probability and semantic entropy). This enables the model to focus on highly relevant products when interests are clear, and to trigger an exploration mechanism when interests are ambiguous, thus alleviating the data sparsity problem in cold start scenarios.

[0023] 4. Adaptive decoding process: By dynamically adjusting the sampling temperature through semantic uncertainty, an adaptive balance between exploration and utilization is achieved under different uncertainty levels.

[0024] 5. Addressing compatibility and scalability: The framework is compatible with different LLM architectures and can be easily extended to handle multimodal inputs through appropriate encoder design, uniformly processing mixed features such as text, categories, and sequences, reducing feature engineering complexity. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a data processing flowchart for a serialization recommendation method based on a large language model in this invention; Figure 2 This is a diagram illustrating the expected performance of an embodiment 2 of a serialization recommendation method based on a large language model in this invention. Figure 3 This is a comparison chart of the expected results of Embodiment 2 of the serialization recommendation method based on a large language model in this invention; Figure 4 This is an example 3 of a serialization recommendation method based on a large language model in this invention, and a diagram showing the expected effect of example 3. Detailed Implementation

[0027] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The technical solution, its implementation process, and principles will be further explained below with reference to the accompanying drawings and specific implementation examples in the embodiments of this application.

[0028] It should be noted that the embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, the present invention covers any substitutions, modifications, equivalent methods and solutions made on the spirit, principles and scope of the present invention as defined by the claims. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] In the description of this application, the terms "first," "second," "third," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "a" or "one," and similar words, do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including," and similar words, mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including," and their equivalents, but do not exclude other elements or objects. The terms "connected" or "linked," and similar words, are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0030] In the description of this application, the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used solely for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, when using positional terms such as "both sides," "outer side," and "upper and lower," it should be understood that they are used only for ease of understanding and description, taking into account that the structure may be oriented to other positions.

[0031] In the description of this application, unless otherwise expressly specified and limited, the technical or scientific terms used shall have the ordinary meaning understood by a person with ordinary skills in the art to which this application pertains. Terms such as “installation,” “connection,” and “joining” shall be interpreted broadly, for example, as fixed connection, detachable connection, mating connection, or integral connection. For a person skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0032] The present invention aims to introduce and explain the structural composition of a serialization recommendation method based on a large language model and the cooperation relationship between the components. Unless otherwise specified, the size, material and manufacturing process of each component in the serialization recommendation method based on a large language model in the present invention can be selected according to specific circumstances, and no special limitations or explanations are made here.

[0033] Furthermore, to provide the public with a better understanding of the present invention, certain specific details are described in detail in the following description of the invention. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0034] Example 1 Please see Figure 1 A serialization recommendation method based on a large language model includes the following steps: A1. Input: Leveraging the powerful sequence modeling capabilities of LLM, we can capture the temporal evolution of interests and input user sets into a large-scale language model (LLM). and item collection .

[0035] For each user Its historical interaction sequence Each of them These are items that have already been interacted with. For example: short-term interests: identifying strong correlations between items at the end of the sequence (e.g., users frequently clicking on electronic products recently); long-term preferences: extracting stable interest patterns from the complete sequence (e.g., users consistently following sports brands).

[0036] Context information of item s For example, attributes such as the item's title, category, brand, and description. Transforming unstructured text into deep semantic representations. For example: title keyword analysis: extracting attributes such as "portability" and "high performance" from "thin and light laptop"; source information fusion: combining structured features such as category and brand to construct a unified semantic representation of items. For example: recommending down jackets instead of short sleeves in winter, incorporating time context; prioritizing local specialty products in tourism scenarios.

[0037] Based on the current user status (such as season and geographical location) and item attributes, it generates scenario-adaptive recommendations. It integrates user behavior time series and item semantic information, breaking through the information limitations of traditional ID-based recommendations, and utilizes the cross-modal understanding capabilities of LLM to process heterogeneous data.

[0038] A2. Data Processing: a) Perform semantic clustering based on the logit representation of candidate items output by a pre-trained large-scale language model (LLM); Furthermore, this includes obtaining the logit vector of candidate items. Calculate the similarity between logit vectors And based on similarity threshold Items are categorized into different semantic clusters.

[0039] Furthermore, the user's historical interaction sequence Contextual information of candidate items The information is input into a pre-trained LLM, which processes it in an autoregressive manner. Its prediction process can be represented as follows: The model generates the logit representation (i.e., the output vector before softmax activation) of a set of candidate items (e.g., K items obtained through preliminary screening or Monte Carlo sampling). Logit vectors, which obtain high-dimensional representations of items in semantic space, preserve the semantic reasoning information of LLM and are more interpretable than traditional embeddings.

[0040] Use the logit vector of candidate items and Calculate the pairwise similarity between them, such as cosine similarity: Semantic equivalence between items is assessed to avoid duplicate recommendations of similar products (e.g., "black T-shirt" and "grey T-shirt" are identified as equivalent).

[0041] Based on the preset similarity threshold This involves grouping items with similar logit vectors (i.e., semantically equivalent items) into the same semantic cluster. Semantic equivalence relations are defined as follows: ,in It is an indicator function. This step divides the candidate item set into several semantic clusters. Candidate items are grouped according to semantic similarity, and redundant candidate sets are compressed, reducing subsequent computational complexity by more than 50% (experimental verification).

[0042] b) Estimating semantic uncertainty based on semantic clustering results; Furthermore, this includes calculating the aggregation probability of each semantic cluster. And calculate semantic entropy based on these cluster probabilities. Aggregate the probability of items within a cluster to reflect the strength of the user's preference for semantic categories, thus solving the problem of dispersed output probabilities in LLM (e.g., the sum of probabilities of 10 similar mobile phones > the probability of a single phone). Furthermore, for each formed semantic cluster Its probability is calculated as the sum of the original probabilities of all items within the cluster: ,in It is an item provided by LLM. The probability of.

[0043] Calculate the overall semantic entropy based on the probability distribution of all semantic clusters. This entropy value quantifies the uncertainty of user preferences at the cluster level.

[0044] c) Decoding steps that adjust item scoring and / or sampling strategies based on semantic uncertainty; Furthermore, the steps for adaptively adjusting item ratings include: combining the item's original probability. The probability of its semantic cluster, the size of the cluster, and the calculated semantic entropy. To generate the final score .

[0045] Furthermore, adaptive scoring: for each candidate item Its final score .in It is the base probability of an item. It's a balancing weight. Function ,in It is an item The semantic cluster to which it belongs It is the size of the cluster. Control the degree of influence of entropy. Balance individual probabilities and semantic cluster weights for dual optimization.

[0046] Furthermore, the steps for adaptively adjusting the sampling strategy include: based on the calculated semantic entropy... Adjusting the sampling temperature during the decoding process Dynamic control exploration - utilizing trade-offs and entropy adaptive mechanisms.

[0047] Furthermore, adaptive sampling temperature: the temperature used for sampling items during the decoding process. .in It is the base temperature. The driving force of uncertainty on temperature is modulated. Higher uncertainty leads to higher sampling temperatures, encouraging exploration; lower uncertainty, on the other hand, causes sampling to focus more on high-probability items.

[0048] Based on the final adaptive score Candidate items are sorted and combined with adaptive sampling temperature. The system selects the top-performing items as the final recommendation. It caters to both real-time prediction (like finding the next song) and list-based recommendations (like an e-commerce homepage). Its end-to-end architecture avoids the accumulation of errors across multiple stages, resulting in minimal response latency.

[0049] A3. Output: Predicted User In its historical interaction sequence The next item most likely to be interacted with. .

[0050] Alternatively, it could be a sorted list of recommended items, representing the final item recommendations.

[0051] Example 2 Please see Figure 1 Application in Amazon product recommendations Background: This example applies a sequential recommendation task to six different domains of an Amazon product dataset: Baby, Beauty, Clothing, Grocery, Sports, and Toys.

[0052] Model backbone: RedPajama-INCITE-Instruct-3B is used as the backbone of a large language model containing 2.8 billion parameters and employing a decoder-only Transformer architecture (32 layers).

[0053] Input: The user's historical interaction sequence, and the item's metadata, such as title, description, category, and brand attributes.

[0054] Framework parameter settings for the method of this invention: semantic similarity threshold Uncertainty weights Uncertainty parameters Number of candidate items sampled Basic sampling temperature Processing flow: User history and candidate item information are entered into the LLM.

[0055] For the Top-K candidate items, this embodiment uses K=10 candidate items to obtain their logit vectors. .

[0056] use Perform logit similarity calculation and semantic clustering.

[0057] Calculate the probability of each semantic cluster and the overall semantic entropy. .

[0058] use and Calculate the final score for each candidate item. .

[0059] Use base temperature and calculated Adjust sampling temperature Then, recommend the next item.

[0060] Expected results: Please refer to Figure 2 , Figure 3 The method of this invention, abbreviated as USD, achieves significant improvements in metrics such as HR@3 (hit rate@3), NDCG@3 (normalized discount cumulative return@3), and MRR@3 (average inverse rank@3) compared to the baseline method, with average improvements of 18.5%, 11.9%, and 10.8%, respectively.

[0061] Example 3 Please see Figure 4 Application in H&M fashion retail recommendations Background: Fashion retail recommendations applied to the H&M dataset, which has a complex retail ecosystem and hierarchical product relationships.

[0062] Continue .

[0063] Processing flow: Similar to Example 2, but the input data is replaced with H&M's user purchase history and product information.

[0064] Expected results: On the H&M dataset, compared with powerful baseline models (such as UniMP), the proposed method USD achieves improvements of 17.6%, 17.5%, and 18.0% in HR@5, NDCG@5, and MRR@5, respectively.

[0065] Example 4 Please see Figure 4 Application in Netflix movie recommendations Background: Movie recommendation applied to the Netflix dataset, which features content-centric preference patterns and temporal dynamics.

[0066] The model backbone and the method of this invention are referred to as the USD framework parameters: (Continued) .

[0067] Processing flow: Similar to Example 2, but the input data is replaced with Netflix user viewing history and movie information.

[0068] Expected results: On the Netflix dataset, compared to UniMP, the proposed method USD achieves improvements of 6.5%, 6.3%, and 6.5% in HR@5, NDCG@5, and MRR@5, respectively.

[0069] These embodiments demonstrate that the method of the present invention, by setting explicit numerical parameters (such as...), This can effectively improve the performance of sequence recommendation on datasets in different domains. These parameters are all within the range given in the technical solution (e.g., Furthermore, experiments have demonstrated the effectiveness of these values. Hyperparameter analysis also shows that, for example... , , The optimal or near-optimal settings observed in the experiment validate the value of the core idea of ​​this invention, namely, that balancing item-specific probabilities and semantic clustering information is crucial for effective recommendations.

[0070] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. It should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A serialization recommendation method based on a large-scale language model, characterized in that, Includes the following steps: A1. Input: Input user set in LLM Item collection and items Context information ; For each user Historical interaction sequence Each of them These are items that have already been interacted with; A2. Data Processing: a) Perform semantic clustering based on the logit representation of candidate items output by a large language model; b) Estimating semantic uncertainty based on semantic clustering results; c) Adjust the decoding steps of item rating and sampling strategy according to semantic uncertainty, and select the items to be recommended as the final recommendation results; A3. Output: Predicting users In its historical interaction sequence The next item most likely to be interacted with. ; Or it could be a sorted list of recommended items, representing the final item recommendations; Step a) includes: User's historical interaction sequence Contextual information of candidate items The information is fed into a pre-trained LLM, which processes it in an autoregressive manner; the LLM generates a logit vector for a set of candidate items. ; Use the logit vector of candidate items and Calculate the pairwise similarity between them: ; Based on the preset similarity threshold Items with similar logit vectors are grouped into the same semantic cluster, and the semantic equivalence relation is defined as follows: ,in It is an indicator function; it divides the candidate item set into several semantic clusters. ; Step b) includes calculating the aggregation probability of each semantic cluster. And calculate semantic entropy based on these cluster probabilities. ; For each formed semantic cluster Its probability is calculated as the sum of the original probabilities of all items within the cluster: ,in It is an item provided by LLM. The probability of; Calculate the overall semantic entropy based on the probability distribution of all semantic clusters. Quantify the uncertainty of user preferences at the cluster level; Adaptive scoring: for each candidate item Its final score ;in It is the original probability of the item. It is a balancing weight; function ,in It is an item The semantic cluster to which it belongs It is the size of the cluster. Control the degree of influence of entropy; The steps for adaptively adjusting the sampling strategy include: based on the calculated semantic entropy Adjusting the sampling temperature during the decoding process ; ;in It is the base temperature. Adjusting the driving force of uncertainty on temperature.

Images