A recommendation method, device and computer program product based on a large language model

Abstract

The application discloses a recommendation method, device and computer program product based on a large language model. The method comprises the following steps: first, obtaining multi-modal interaction information such as text information, image information and video information related to a target user; then, using an LLM, performing iterative implicit feedback enhancement processing on the text information, and / or performing item attribute enhancement processing based on contrastive semantic enhancement and user portrait enhancement processing based on contrastive semantic enhancement, to obtain enhanced text information. Next, a text feature vector of the enhanced text information, an image feature vector of the image information and a video feature vector of the video information are generated, and then the three feature vectors are subjected to optimization processing of semantic alignment based on a view-text contrast learning strategy. The three optimized feature vectors are subjected to fusion processing, and then the fused multi-modal unified feature vector is input into a recommendation model to obtain a more accurate item recommendation result for the target user, so that an ideal recommendation effect is achieved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a recommendation method, device, and computer program product based on a large language model. Background Technology

[0002] With the rapid development of internet platforms, recommendation systems have become an indispensable key component in various application scenarios, widely used in e-commerce, video streaming platforms, and social media, providing various conveniences for people's lives.

[0003] Currently, in existing recommender systems, the classic architecture of "multi-encoder + fusionist" is commonly used to ensure semantic consistency across different modalities within a unified space, achieving unified semantic representation of multiple modalities. However, while this architecture can achieve preliminary fusion of features from different modalities to a certain extent, its effectiveness in cross-modal alignment is still insufficient, limiting the recommendation results. Meanwhile, data sparsity remains a challenge in existing recommender systems. Current methods address this by introducing auxiliary information. However, this approach often brings a series of negative effects, such as introducing noise interference, causing usability issues, and generating low-quality data. These problems, in turn, hinder the accurate modeling of user preferences, directly affecting the accuracy and personalization of the recommendation results. Therefore, the recommendation methods used in existing recommender systems all have significant problems and struggle to achieve ideal recommendation results. Summary of the Invention

[0004] The main objective of this application is to provide a recommendation method, device, and computer program product based on a large language model (LLM), which can improve the efficiency and accuracy of item recommendations for users based on the large language model (LLM) to achieve ideal recommendation results and thus improve user experience.

[0005] This application provides a recommendation method based on a large language model, including:

[0006] Acquire multimodal interaction information related to the target user; the multimodal interaction information includes text information, image information, and video information;

[0007] Using a large language model, iterative implicit feedback enhancement processing is performed on the text information, and / or, using a large language model, project attribute enhancement processing based on contrastive semantic enhancement and user profile enhancement processing based on contrastive semantic enhancement are performed on the text information to obtain enhanced text information;

[0008] Generate the text feature vector corresponding to the enhanced text information, the image feature vector corresponding to the image information, and the video feature vector corresponding to the video information;

[0009] Based on the image-text contrast learning strategy, the text feature vector, image feature vector, and video feature vector are semantically aligned and optimized; and the optimized text feature vector, optimized image feature vector, and optimized video feature vector are fused to obtain a multimodal unified feature vector.

[0010] The multimodal unified feature vector is input into the recommendation model to obtain the recommendation results for the target user.

[0011] In one possible implementation, the iterative implicit feedback enhancement processing of text information using a large language model includes:

[0012] Using the text information and the information of the project to be recommended, a first prompt instruction is generated and input into the large language model to obtain positive and negative sample data selected by the model, which constitutes sample training data.

[0013] The sample training data and the pre-acquired original training data are merged into recommendation data; and the recommendation system is iteratively trained using the recommendation data to obtain enhanced text information after training is completed.

[0014] In one possible implementation, the use of a large language model to perform contrastive semantic enhancement on text information for both item attribute enhancement and user profile enhancement includes:

[0015] Using the text information, a second prompt instruction is generated, input into the large language model, and auxiliary information output by the model is obtained; and the auxiliary information is used to generate preliminarily enhanced text information.

[0016] The text information and the pre-enhanced text information are respectively input into a pre-trained language model for encoding to generate a first encoding vector corresponding to the text information and a second encoding vector corresponding to the pre-enhanced text information.

[0017] The first and second coding vectors are normalized to obtain the normalized first coding vector and the normalized second coding vector.

[0018] Contrastive learning training is performed using a loss function based on noise contrast estimation to narrow the distance between the normalized first encoding vector and the normalized second encoding vector until a preset condition is met. At this point, training stops, and the enhanced text information that has completed the contrastive learning training is obtained.

[0019] In one possible implementation, generating the text feature vector corresponding to the enhanced text information, the image feature vector corresponding to the image information, and the video feature vector corresponding to the video information includes:

[0020] The enhanced text information is encoded using the BGE-m3 model to obtain text feature vectors; and the image information and video information are encoded using the GME model to obtain image feature vectors and video feature vectors, respectively.

[0021] In one possible implementation, the optimization process for semantic alignment of the text feature vector, image feature vector, and video feature vector based on the view-text contrastive learning strategy includes:

[0022] Calculate the similarity between image feature vectors and text feature vectors, and the similarity between text feature vectors and video feature vectors; and use the obtained similarity to calculate the value of the view-text comparison loss function until the value meets the preset condition, then stop the semantic alignment optimization processing of the text feature vectors, image feature vectors, and video feature vectors, to obtain the optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors.

[0023] In one possible implementation, the fusion processing of the obtained optimized text feature vector, optimized image feature vector, and optimized video feature vector to obtain a multimodal unified feature vector includes:

[0024] Through a cross-attention mechanism, the optimized text feature vector, optimized image feature vector, and optimized video feature vector are input into the W-layer Transformer structure for deep interaction and fusion layer by layer to obtain a multimodal unified feature vector; where W is a positive integer greater than 0.

[0025] In one possible implementation, the recommendation model is based on the group relative policy optimization GRPO method, which is obtained by training a large language model in two stages: supervised fine-tuning and reinforcement learning, using supervised fine-tuning datasets and reinforcement learning datasets.

[0026] This application also provides a recommendation device based on a large language model, including: a processor, a memory, and a system bus;

[0027] The processor and the memory are connected via the system bus;

[0028] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the above-described implementations of the recommendation method based on the large language model.

[0029] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any of the above-described implementations of the recommendation method based on a large language model.

[0030] This application provides a recommendation method, device, and computer program product based on a large language model. First, it acquires multimodal interaction information related to the target user; this multimodal interaction information includes text, image, and video information. Then, using a large language model, iterative implicit feedback enhancement processing is applied to the text information, and / or, using the large language model, it performs item attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement, resulting in enhanced text information. Next, text feature vectors corresponding to the enhanced text information, image feature vectors corresponding to the image information, and video feature vectors corresponding to the video information are generated. Then, based on a view-text contrastive learning strategy, semantic alignment optimization processing is performed on the text feature vectors, image feature vectors, and video feature vectors. The optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors are fused to obtain a unified multimodal feature vector. This unified multimodal feature vector can then be input into a recommendation model to obtain the model's output recommendation result for the target user.

[0031] As can be seen, when recommending items to target users, this application first uses a large language model to perform iterative implicit feedback enhancement processing on text information related to target users, and / or item attribute enhancement processing based on contrastive semantic enhancement and user profile enhancement processing based on contrastive semantic enhancement to obtain enhanced text information. This can solve the problem of noise interference in implicit feedback caused by data sparsity in the prior art, making it more accurately reflect the true preferences of target users, and also enhancing the robustness of the model in noisy environments.

[0032] Building upon this foundation, this application employs a text-image contrastive learning strategy to achieve accurate modeling of multi-source heterogeneous data, including text, images, and videos (specifically text feature vectors, image feature vectors, and video feature vectors), within a unified representation space. This yields a unified multimodal feature vector, which serves as the basis for recommendations. This vector is then input into the recommendation model to obtain recommendations tailored to the target user. This approach avoids the information loss caused by modal fragmentation in existing recommendation systems, improves the efficiency and accuracy of item recommendations for the target user, achieves the desired recommendation effect, and enhances the user experience. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a recommendation method based on a large language model, provided for an embodiment of this application;

[0035] Figure 2 A schematic diagram of the overall recommendation process based on a large language model provided in this application embodiment;

[0036] Figure 3 This is a schematic diagram illustrating the process of iterative implicit feedback enhancement of text information using a large language model, as provided in an embodiment of this application.

[0037] Figure 4 This is a schematic diagram illustrating the process of using a large language model to perform project attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement on text information, as provided in the embodiments of this application.

[0038] Figure 5 A schematic diagram illustrating the training process of the recommendation model provided in this application embodiment;

[0039] Figure 6 This is a schematic diagram of the structure of a recommendation device based on a large language model, provided as an embodiment of this application. Detailed Implementation

[0040] With the rapid development of internet platforms, recommendation systems have become an indispensable key component in various application scenarios, widely used in e-commerce, video streaming platforms, and social media, providing various conveniences for people's lives.

[0041] However, existing recommendation systems also face some fundamental challenges and limitations. Specifically, the increasing complexity of user behavior and content formats means that recommendation systems must not only properly handle traditional user-item interactions (such as user purchase behavior and user video viewing history), but also fully integrate data from multiple modalities, including text, images, and videos. Text information can refer to user comments on social media platforms, product descriptions, etc.; image information can include, but is not limited to, product images and scene shots in videos; and audio information can refer to the video content watched by users.

[0042] However, existing recommender systems generally employ a "multi-encoder + fusioner" architecture. While this achieves some initial fusion of features from different modalities, its effectiveness in cross-modal alignment remains insufficient. Especially in semantic interactions involving video, images, and text, information fragmentation easily occurs, preventing the full exploitation of intermodal relationships. Therefore, significant deficiencies remain in the unification of multimodal representations, limiting recommender performance. Simultaneously, data sparsity remains a challenge in recommender systems. Existing methods address this by introducing auxiliary information. However, this approach often introduces negative effects, such as noise interference, usability issues, and the generation of low-quality data. These problems, in turn, hinder accurate modeling of user preferences, directly impacting the accuracy and personalization of recommendation results.

[0043] Specifically, existing recommendation systems typically suffer from the following two shortcomings:

[0044] (1) Insufficient construction and alignment of multimodal unified representation.

[0045] Specifically, in the modal information processing of existing recommender systems, to ensure semantic consistency across different modalities within a unified space, multimodal unified semantic representation models typically adopt the classic architecture of "multi-encoder + fusionist," designing a dedicated feature extractor for each modality. However, it is difficult to effectively align features from different modalities within a unified space. Existing methods often use image-text pairs as positive samples and random image-text pairs as negative samples, introducing multi-task joint training strategies. However, these methods suffer from modal fragmentation and representation limitations. Specifically, they lack a unified representation framework for video-image-text trimodal representation, resulting in low efficiency in cross-modal semantic transfer. The temporal dynamic information of videos and the spatial details of images fail to complement each other, reducing the recommender model's ability to deeply understand diverse content.

[0046] (2) Insufficient implicit feedback and auxiliary information.

[0047] In practical applications of existing recommender systems, they rely not only on interactive and collaborative information between users and items but also heavily on auxiliary content such as text, images, and videos. The presence of this auxiliary information can alleviate, to some extent, the problem of insufficient modeling of user-item relationships caused by sparse interactions. However, it's important to note that the introduction of auxiliary information inevitably introduces noise. Specifically, potential deficiencies in the completeness and quality of auxiliary information can negatively impact the accuracy of user profiling. Furthermore, current data augmentation methods still face challenges in handling implicit feedback, including false positives (which can be understood as accidental clicks, popularity bias, etc.) and false negatives (e.g., no interaction ≠ dislike). False positives in implicit feedback mainly manifest as unreliable user-item interactions. This includes items that users interact with not out of genuine intent, such as interactions caused by accidental clicks or popularity bias. False negatives, on the other hand, are caused by non-interactive items, which may not actually represent user dislike but are often treated as negative samples in practice.

[0048] Furthermore, in previous recommendation systems that used auxiliary information, the auxiliary information usually existed in the form of item attributes, which had problems such as incomplete information, low quality, and difficulty in obtaining it. The generated pseudo samples had weak semantic relevance and could not effectively supplement the recommendation system. Therefore, the noise interference in the implicit feedback made it difficult to accurately capture the user's true intention, and the low quality of the auxiliary information further aggravated the decline in the model's generalization ability.

[0049] It is evident that the recommendation methods used in existing recommendation systems all have significant problems and are difficult to achieve the desired recommendation results.

[0050] To address the aforementioned shortcomings, this application provides a recommendation method based on a large language model. First, it acquires multimodal interaction information relevant to the target user, including text, image, and video information. Then, using a large language model, iterative implicit feedback enhancement processing is applied to the text information, and / or, the large language model is used to perform item attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement, resulting in enhanced text information. Next, text feature vectors corresponding to the enhanced text information, image feature vectors corresponding to the image information, and video feature vectors corresponding to the video information are generated. Then, based on a view-text contrastive learning strategy, semantic alignment optimization processing is performed on the text feature vectors, image feature vectors, and video feature vectors. Finally, the optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors are fused to obtain a unified multimodal feature vector. This unified multimodal feature vector can then be input into a recommendation model to obtain the model's output recommendation result for the target user.

[0051] As can be seen, when recommending items to target users, this application first uses a large language model to perform iterative implicit feedback enhancement processing on text information related to target users, and / or item attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement, to obtain enhanced text information. This solves the problem of noise interference in implicit feedback caused by data sparsity in the prior art, making it more accurately reflect the true preferences of target users and enhancing the robustness of the model in noisy environments.

[0052] Building upon this foundation, this application employs a text-image contrastive learning strategy to achieve accurate modeling of multi-source heterogeneous data, including text, images, and videos (specifically text feature vectors, image feature vectors, and video feature vectors), within a unified representation space. This yields a unified multimodal feature vector, which serves as the basis for recommendations. This vector is then input into the recommendation model to obtain recommendations tailored to the target user. This approach avoids the information loss caused by modal fragmentation in existing recommendation systems, improves the efficiency and accuracy of item recommendations for the target user, achieves the desired recommendation effect, and enhances the user experience.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] See Figure 1 This is a schematic diagram of a recommendation process based on a large language model provided in this embodiment. The method includes the following steps:

[0055] S101: Obtain multimodal interaction information related to the target user; wherein, the multimodal interaction information includes text information, image information and video information.

[0056] It should be noted that with the continuous development of artificial intelligence and information technology, various large language models (LLMs) have emerged in people's lives and work, providing intelligent interactive functions for various application scenarios such as knowledge question answering. The emergence of large models has enabled a qualitative leap in understanding text processing logic tasks. Therefore, this application proposes a method for intelligent recommendation based on large language models. By combining data augmentation techniques such as image-text comparison learning, multimodal matching and discrimination, and iterative implicit feedback, the alignment effect of multimodal unified representations is improved, effectively solving the data sparsity problem. This ensures that the user profiles generated by the large language model and the item attributes maintain good distinguishability and high relevance in the embedding space, thereby providing strong support for the performance improvement of the recommendation system, improving the efficiency and accuracy of item recommendations for users, achieving ideal recommendation results, and improving user experience.

[0057] In this embodiment, the method provided in this application is used to define any user targeted when making project recommendations based on a large language model as a target user. In order to improve the efficiency and accuracy of project recommendations for target users and improve the target user experience, multimodal interaction information related to the target user can be obtained from different data sources, including but not limited to text information, image information and video information, as a basis for recommendation and to execute subsequent recommendation steps.

[0058] This application does not limit the methods for acquiring multimodal interaction information such as text, image, and video information, nor the specific content included. For example, if a target user browsed multiple products and watched related video content in the past week, then the text information related to the target user could come from the product titles and descriptions; the image information could be product images; and the video information could include the video content watched by the target user. By collecting these different types of interaction data, a rich information foundation can be provided for the subsequent recommendation process.

[0059] S102: Using a large language model, perform iterative implicit feedback enhancement processing on the text information, and / or, using a large language model, perform item attribute enhancement processing based on contrastive semantic enhancement and user profile enhancement processing based on contrastive semantic enhancement on the text information to obtain enhanced text information.

[0060] In this embodiment, after obtaining the text information generated by the target user during historical interactions in step S101, further, based on the data augmentation strategy of LLM, iterative implicit feedback augmentation processing can be performed on the text information using LLM, and / or item attribute augmentation processing driven by Contrastive Semantic Augmentation (CSA) and user profile augmentation processing driven by CSA can be performed to obtain the augmented text information, such as... Figure 2 As shown, this effectively mitigates the noise interference problem of implicit feedback caused by data sparsity.

[0061] Specifically, one possible implementation is to utilize a large language model. The iterative implicit feedback enhancement process for text information can include: first, integrating the content of the text information and the information of the item to be recommended into a prompt word to generate a first prompt instruction, which is then input into the large language model (the specific composition structure is not limited and can be selected according to the actual situation and experience, such as the iFlytek large model), to obtain positive and negative sample data selected by the model, which constitute the sample training data (which can be represented as...). Then the sample training data () The system combines the original training data (represented as E, which typically refers to a dataset that has been collected and prepared without any data augmentation, such as user-item interaction records (e.g., a target user purchasing item X), which may include positive samples (the target user is interested in an item) and negative samples (the target user did not interact with an item)) into recommendation data. This recommendation data is then used to iteratively train the recommendation system to obtain augmented text information after training.

[0062] It's important to note that this application employs LLM for data augmentation precisely to leverage its powerful reasoning capabilities and knowledge support, using it as a knowledge-aware sampler to augment the training data for pairwise Bayesian Personalized Ranking (BPR). The core idea of ​​BPR is to optimize the recommendation system's performance by learning how to rank items based on users' historical interaction data. More specifically, given a set of items (e.g., A and B), the BPR model aims to learn "which item is more attractive to the user." It trains the model by learning the ranking relationship between pairs of positive and negative samples (i.e., items the user has interacted with and items they haven't). Positive samples: user interactions with items, indicating the user's interest in the item (e.g., clicks, purchases). Negative samples: no user interaction with items, indicating the user's lack of interest in the item. By ranking this data, BPR optimizes how the model infers the user's preferences for other unseen items based on these positive and negative sample pairs.

[0063] In this implementation, a first prompt instruction can be constructed for each target user. This first prompt instruction consists of each target user's historical interaction records (specifically text information) and candidate items (i.e., information on items to be recommended). This allows the target user's historical interaction items and candidate item pool to be combined. The input is an LLM (Local Recommendation Model). Since the LLM cannot globally rank all items, this application introduces a basic recommendation model to filter out "hard samples" with high prediction scores (the specific values ​​are not limited; for example, the values ​​can be the prediction scores given by the recommendation model in the range [0, 1]) as a candidate set. These can include, but are not limited to, high residual samples (samples with large prediction errors, i.e., samples where the prediction results of the recommendation system differ significantly from the user's actual behavior) and uncertain samples (items that the recommendation system cannot clearly classify as positive or negative samples. For example, a user has interacted with an item, but the intensity of the interaction is very low, and the system cannot determine whether it is a positive sample). The valuable positive and negative sample data that the LLM selects constitute the sample training data. Thus, the LLM can then... Select items that the target user might like (represented as...) ) and items you don't like (indicated as) That is, each user selects a positive sample and a negative sample as pseudo-BPR training data.

[0064] Then, after inputting the constructed first prompt command into LLM, a certain amount of data can be selected from the generated training data (i.e., Then, it is merged with the original training data (E). The specific calculation formula is as follows:

[0065]

[0066]

[0067] in, Represents the set of pairwise triples generated by LLM; This indicates the first prompt instruction, which includes a task description, user interaction history, a list of candidate items, and output format instructions.

[0068] Based on this, the final merged recommendation data For training the recommendation system, after t (t=0,...,T-1) training rounds, high-value candidate items are re-collected based on the current prediction error of the model, new enhanced interaction data is iteratively generated, and then the prediction error (denoted as ) for each (target) user-item pair (u,i) is calculated on the validation set. And construct a candidate set of high residuals, the specific calculation formula is as follows:

[0069]

[0070]

[0071] in, Let represent the representation vectors of user u and project i after t rounds of training; Represents the set of real positive interactions. This indicates an indicator function if (u,i) lies in the true positive interaction set. If the condition is met, the function value is 1; otherwise, it is 0. The true positive interaction set... This refers to the actual set of interactions between users and projects in practice. It includes positive interactions (users have already interacted with the project, such as clicking, purchasing, rating, etc.) and negative interactions (users have not interacted with the project, or have explicitly expressed disinterest (e.g., no clicks, no purchases, etc.)). It should be noted that for each (target) user, this can be derived from all "high residual" non-interaction pairs in round t. Selecting Top K as the candidate pool This is used to construct the prompt instruction information. . This represents the residual threshold. New training data consisting of positive and negative samples is generated by calling LLM. Then, the edge set for the next round of training can be updated. The iteration terminates when the performance converges at t=T. Performance convergence means that the model's performance no longer significantly improves after multiple training rounds. This is typically reflected in the prediction error on the validation set no longer decreasing or fluctuating very little, indicating that the model training is nearing completion, so the iteration can be terminated at this point. If the maximum number of training rounds is set to T=50, training will terminate after 50 rounds, or earlier if the model performance converges (e.g., the validation error no longer significantly decreases). The specific calculation formula is as follows:

[0072]

[0073]

[0074] in, Let represent the set of triples in round t; Let represent the set of triples in round t+1; This represents the data after the (t+1)th training round. The calculation method can be found in the formula above. The iteration of "". Indicates noise trimming (e.g.) Figure 3 The operation (shown) removes triples with low confidence or large gradient perturbations. By using LLM as an implicit feedback augmenter, this application leverages the powerful inference capabilities and knowledge support of LLM to acquire more meaningful and information-rich samples. After each training round, high-value candidates are re-collected based on the model's current prediction error, iteratively generating new augmented data to effectively alleviate the problem of diminishing value of initially generated augmented sample data in later training.

[0075] Here, "low confidence level" usually refers to a higher uncertainty in the model's predictions for certain samples. This can be determined in the following ways:

[0076] A predicted value close to 0.5 (for binary classification problems) or close to the median (for regression problems) means that the model is uncertain about the classification or prediction of that sample.

[0077] "Large gradient perturbation" typically refers to situations where, during model training, the gradient update magnitude for certain samples is extremely large, potentially leading to training instability. This can be determined using the following methods:

[0078] The absolute value of the gradient is large (e.g., greater than a certain threshold).

[0079] The gradient changes significantly, which means that the model parameters are updated too quickly during training, resulting in excessive error fluctuations.

[0080] Another optional implementation is to utilize a large language model to perform contrastive semantic enhancement on both item attribute enhancement and user profile enhancement. This process can include: first, incorporating the content of the text information into a prompt word to generate a second prompt instruction, which is then input into the large language model to obtain auxiliary information output by the model; and finally, using this auxiliary information to generate preliminarily enhanced text information (which can be represented as...). Then, the text information and the initially enhanced text information are input into the pre-trained language model for encoding, generating a first encoded vector corresponding to the text information and a second encoded vector corresponding to the initially enhanced text information. Next, the first and second encoded vectors can be normalized to obtain normalized first and second encoded vectors. Then, contrastive learning training is performed using a loss function based on noise contrastive estimation (InfoNCE) to narrow the distance between the normalized first and second encoded vectors until a preset condition is reached (the specific content is not limited and can be set according to the actual situation and experience, such as setting the loss function value to be so small as to remain basically unchanged, or reaching a preset number of training times (such as 100 times)). At this point, training stops, and the enhanced text information that has completed the contrastive learning training is obtained.

[0081] In this implementation, a second cue instruction for data augmentation is first constructed for each target user / project. Then, the generated second cue instruction (cue word) is input into the LLM to generate additional auxiliary information for each target user / project. The generated augmented attributes (i.e., such as...) are then... Figure 4 shown ) and primitive attributes (which can be represented as (U,I), such as Figure 4 The "raw data" shown are all input into a pre-trained language model with encoding capabilities for encoding to obtain embedding vectors. ,like Figure 4 The diagram shows the "original vector" and "enhanced vector". Finally, the generated encoded vectors are normalized, and the InfoNCE method of contrastive learning (i.e., by calculating the value of the loss function) is used to narrow the distance between the original attribute vector and the enhanced attribute vector of each entity. The specific calculation formula is shown below:

[0082]

[0083]

[0084]

[0085] in, Represents the contrastive loss function; User information u and user prompt word P U Similarly, the concatenated user data enhancement prompts, This indicates the project enhancement prompt; N represents the number of samples participating in the comparison; f represents the "original vector"; Indicates "enhanced vector"; The value represents the temperature coefficient for comparative learning; its specific value is not limited and can be set according to actual conditions and experience. The values ​​< and > represent the calculation of the inner product of the original vector and the augmented vector.

[0086] S103: Generate the text feature vector corresponding to the enhanced text information, the image feature vector corresponding to the image information, and the video feature vector corresponding to the video information.

[0087] In this embodiment, after obtaining the text, image, and video information generated by the target user during historical interactions in step S101, and obtaining the enhanced text information in step S102, existing or future feature vector generation methods can be used to generate text feature vectors corresponding to the enhanced text information (such as...). Figure 2 The “text vector” shown can be represented as ,), and the image feature vector corresponding to the image information (such as ,), Figure 2 The "image vector" shown can be represented as ,), and the video feature vector corresponding to the video information (such as ,), Figure 2 The “video vector” shown can be represented as This is used to execute the subsequent step S104. Wherein, , , It is an embedding of the [CLS] tag.

[0088] Specifically, one possible implementation is to utilize the BGE-m3 model as a text encoder, leveraging its powerful cross-language and cross-domain text representation capabilities to encode the enhanced text information and obtain text feature vectors. Furthermore, the General Multimodal Embedding (GME) model can be used to encode image and video information separately, obtaining image and video feature vectors. Specifically, for image information, spatial static features (i.e., image feature vectors) can be extracted using a convolutional + Transformer cascade structure; for video information, dynamic evolution information can be captured using temporal convolution combined with an attention mechanism, serving as video feature vectors.

[0089] S104: Based on the image-text contrast learning strategy, semantic alignment optimization is performed on text feature vectors, image feature vectors, and video feature vectors; and the optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors are fused to obtain a multimodal unified feature vector.

[0090] In this embodiment, after obtaining the text feature vector, image feature vector, and video feature vector through step S103, further, firstly, based on the view-text comparison learning strategy, semantic alignment optimization processing can be performed on the text feature vector, image feature vector, and video feature vector to obtain optimized text feature vector, optimized image feature vector, and optimized video feature vector. Specifically, the similarity between the image feature vector and the text feature vector, and the similarity between the text feature vector and the video feature vector can be calculated first; then, the value of the view-text comparison loss function can be calculated using the obtained similarity, until the value meets the preset conditions (the specific content is not limited and can be set according to the actual situation and experience, such as setting the value of the view-text comparison loss function to be small enough to remain basically unchanged, or reaching 100 training times, etc.), then the semantic alignment optimization processing of the text feature vector, image feature vector, and video feature vector is stopped, resulting in optimized text feature vector, optimized image feature vector, and optimized video feature vector. Then, through the cross-attention mechanism, the optimized text feature vector, optimized image feature vector, and optimized video feature vector are input into the W (the specific value is not limited and can be any positive integer greater than 0) layer Transformer structure for deep interaction and fusion layer by layer, so as to achieve the complementary advantages of temporal dynamic information and spatial static features, and obtain a multimodal unified feature vector, which is used to execute the subsequent step S105.

[0091] It should be noted that this application proposes a view-text contrastive learning strategy precisely to optimize multimodal alignment, thereby improving the representation quality of each modality (such as video, image, and text) before feature vector fusion. The core objective of view-text contrastive learning is to ensure that text feature vectors, image feature vectors, and video feature vectors achieve semantic alignment before fusion, ensuring that each modality has a common semantic representation in the feature space.

[0092] In practical applications, the specific implementation process can be as follows: First, calculate the similarity between the image feature vector and the text feature vector; that is, the image feature vector and the text feature vector will be aligned through contrastive learning. The image feature vector serves as the query, and the text feature vector serves as the key. Calculate the similarity between them and represent it as... Conversely, using the text feature vector as the query and the image feature vector as the key, the similarity between them is calculated and represented as... Similarly, using video feature vectors as queries and text feature vectors as keys, the similarity between them is calculated and represented as... Conversely, using the text feature vector as the query and the video feature vector as the key, the similarity between them is calculated and represented as... Then, a preferred implementation is to use the Softmax function to convert the similarity (specifically the corresponding score) into a probability distribution (which can be represented as...). The specific calculation formula is as follows:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] in, , and These represent linear transformations that map [CLS] embeddings to a normalized low-dimensional space. The temperature coefficient is used to adjust the smoothness of the output probability. Its specific value is not limited and can be set according to actual conditions and experience; for example, it can be 0.1. M represents the total number of mode pairs. Based on this, the corresponding cross-entropy loss is calculated (which can be expressed as...). The formula for calculating () can be shown below:

[0099]

[0100] in, The true label is represented by 1, indicating a match, and 0, indicating a non-match; P represents the number of image or video samples; Q represents the number of text samples, thus yielding the final total loss (which can be expressed as...). ) can be:

[0101] +

[0102] In addition, during data processing, 15% (this is just an example value and is not a limitation) of the input text tokens were randomly selected and replaced with [MASK] to predict the distribution of masked words. The specific calculation formula is as follows:

[0103]

[0104] in, The loss function represents the modeling loss function for masked language; k represents the set of mask position indices. Indicates contextual vocabulary; The representation model represents the masked words. The predicted probability.

[0105] For matching between videos, images, and text, this application utilizes the [CLS] features output by a multimodal encoder (the specific composition is not limited, but a 6-layer Transformer structure can be used, with its initial weights taken from the parameters of the last 6 layers of the BERT-base model), and adds a fully connected network for binary classification prediction. The corresponding calculation formula is as follows:

[0106]

[0107] in, The loss function for visual-text matching; This represents a binary classification label. Positive sample pairs are represented by 1, and negative sample pairs by 0. To further improve matching quality, this application samples "difficult negative samples" from mini-batch data based on the contrastive similarity distribution, and instead of randomly sampling negative samples, it samples based on the similarity distribution calculated during the ITC stage. This means that samples with high similarity (semantic similarity) in the feature space but not actually matching (e.g., "husky in the snow" vs. "husky in the grass") are more likely to be selected as negative samples. This forces the model to learn fine-grained, detailed differences between these samples, which are semantically similar but differ in detail.

[0108] Based on this, the final loss function is obtained by weighted summing of the loss functions of the three tasks mentioned above. This final loss function serves as the image-text comparison loss function and is expressed as follows: This enables end-to-end joint training, and the specific calculation formula is as follows:

[0109]

[0110] in, These represent the hyperparameter weights of each loss function, used to balance the contributions of contrastive learning, generation tasks, and discrimination tasks during training. They are typically initially set to 1.0 and can be adjusted as needed.

[0111] Next, through a cross-attention mechanism, the optimized text feature vector, image feature vector, and video feature vector are input into a multi-layer Transformer structure for deep interaction and fusion layer by layer. In short, this mechanism acts as an information "bridge," automatically adjusting feature weights in each layer based on the correlation between modalities. This efficiently completes the transmission and integration of cross-modal information, achieving complementary advantages between temporal dynamic information and spatial static features, resulting in a more accurate unified multimodal feature vector. This is also thanks to the thorough optimization of the front-end model parameters and the deep fusion of the back-end, ensuring that the final output feature vector is no longer a simple superposition of isolated modal information, but a high-confidence representation that has undergone semantic disambiguation and contextual alignment. This corrects biases from a single-modal perspective and locks in the unique true semantics in a multimodal environment.

[0112] S105: Input the multimodal unified feature vector into the recommendation model to obtain the recommendation results for the target user.

[0113] In this embodiment, after obtaining the multimodal unified feature vector through step S104, the multimodal unified feature vector can be further input into the recommendation model so that the model can combine with the recommendation system to predict more accurate item recommendation results for the target user, thereby improving the target user experience.

[0114] It should be noted that the specific composition and training process of the recommendation model can be set according to the actual situation. This embodiment does not limit this. One possible implementation is that the recommendation model can be based on the Group Relative Policy Optimization (GRPO) method, and is obtained by training a large language model in two stages of supervised fine-tuning and reinforcement learning using supervised fine-tuning (SFT) dataset and reinforcement learning (RL) dataset.

[0115] In this implementation, such as Figure 5As shown, the collected dataset can first be divided into SFT datasets and RL datasets. For example, the SFT dataset may contain data such as: "{"messages": [{"role": "user", "content": "## Please determine whether this group will watch the following self-media videos based on the group's historical interaction information:\n## Group type:\nworker\n\n## Historical interaction information:\nTitle: \"xxx\", Video type: \"[Food]\", Video platform: \"xx\",\nTitle: \"xxx\"……}". The RL dataset may contain data such as: "{"messages": [{"role": "user", "content": "## Please determine whether this group will watch the following self-media videos based on the group's historical interaction information:\n## Group type:\nworker\n\n## Historical interaction information:\nTitle: \"……}" etc.

[0116] Then, during the SFT cold start phase, the large language model LLM (such as generative large models) can be supervised and fine-tuned using a small amount of high-quality interaction data from the SFT dataset, giving it basic recommendation sub-task capabilities, ensuring that the model can generate query rewriting or intent summaries that meet basic quality requirements, and completing the first stage of model training.

[0117] Next, after entering the policy gradient reinforcement learning stage, as follows: Figure 5 As shown, a closed-loop optimization mechanism can be used to tightly connect large language models (LLMs, such as generative large models) with downstream black-box recommendation systems. Specifically, the feedback signals from the recommendation system can be directly used to optimize the generation strategy of the large language model (LLM): the large language model generates text outputs (such as rewritten queries, user profile descriptions, or item feature descriptions) based on inputs (such as user queries or behavioral history). The recommendation model uses task-specific evaluation metrics to measure the quality of these text outputs and converts them into scalar reward signals, driving the reinforcement learning process to optimize the policy parameters of the large language model (LLM). This application employs the Group Relative Policy Optimization (RPO) algorithm for optimization, avoiding the high memory consumption problem of traditional reinforcement learning algorithms during training, thus effectively bridging the gap between generation quality and recommendation performance, while significantly reducing the dependence on manual annotation or APIs for the large language model (LLM).

[0118] In summary, this embodiment provides a recommendation method based on a large language model. First, it acquires multimodal interaction information related to the target user, including text, image, and video information. Then, using a large language model, iterative implicit feedback enhancement processing is applied to the text information, and / or, the large language model is used to perform item attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement, resulting in enhanced text information. Next, text feature vectors corresponding to the enhanced text information, image feature vectors corresponding to the image information, and video feature vectors corresponding to the video information are generated. Then, based on a view-text contrastive learning strategy, semantic alignment optimization processing is performed on the text feature vectors, image feature vectors, and video feature vectors. Finally, the optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors are fused to obtain a unified multimodal feature vector. This unified multimodal feature vector can then be input into the recommendation model to obtain the model's output recommendation result for the target user.

[0119] As can be seen, when recommending items to target users, this application first uses a large language model to perform iterative implicit feedback enhancement processing on text information related to target users, and / or item attribute enhancement processing and user profile enhancement processing based on contrastive semantic enhancement, to obtain enhanced text information. This solves the problem of noise interference in implicit feedback caused by data sparsity in the prior art, making it more accurately reflect the true preferences of target users and enhancing the robustness of the model in noisy environments.

[0120] Building upon this foundation, this application employs a text-image contrastive learning strategy to achieve accurate modeling of multi-source heterogeneous data, including text, images, and videos (specifically text feature vectors, image feature vectors, and video feature vectors), within a unified representation space. This yields a unified multimodal feature vector, which serves as the basis for recommendations. This vector is then input into the recommendation model to obtain recommendations tailored to the target user. This approach avoids the information loss caused by modal fragmentation in existing recommendation systems, improves the efficiency and accuracy of item recommendations for the target user, achieves the desired recommendation effect, and enhances the user experience.

[0121] Furthermore, embodiments of this application also provide a recommendation device based on a large language model, including: a processor, a memory, and a system bus;

[0122] The processor and the memory are connected via the system bus;

[0123] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the above-described implementations of the recommendation method based on the large language model.

[0124] refer to Figure 6 The diagram illustrates a suitable structure for implementing the large language model-based recommendation device in the embodiments of this application. The large language model-based recommendation device in the embodiments of this application may include, but is not limited to, terminals or servers equipped with a large language model. Figure 6 The recommendation device based on a large language model shown is merely an example of an electronic device and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 6 As shown, the large language model-based recommendation device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. When the large language model-based recommendation device is powered on, RAM 603 also stores various programs and data required for device operation. The processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0126] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, memory card, hard disk, etc.; and communication devices 609. Communication device 609 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Recommended devices based on a large language model with various means are shown; however, it should be understood that implementation or possession of all the means shown is not required. More or fewer means may be implemented alternatively.

[0127] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the recommendation method based on a large language model.

[0128] Furthermore, this application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any of the above-described implementations of the recommendation method based on a large language model.

[0129] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0130] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0131] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A recommendation method based on a large language model, characterized in that, include: Acquire multimodal interaction information related to the target user; The multimodal interaction information includes text information, image information, and video information; Using a large language model, iterative implicit feedback enhancement processing is performed on the text information. The large language model is also used to perform project attribute enhancement processing based on contrastive semantic enhancement and user profile enhancement processing based on contrastive semantic enhancement, resulting in enhanced text information. Generate the text feature vector corresponding to the enhanced text information, the image feature vector corresponding to the image information, and the video feature vector corresponding to the video information; Based on the image-text contrast learning strategy, the text feature vector, image feature vector, and video feature vector are semantically aligned and optimized; and the optimized text feature vector, optimized image feature vector, and optimized video feature vector are fused to obtain a multimodal unified feature vector. The multimodal unified feature vector is input into the recommendation model to obtain the recommendation result for the target user output by the model. The method of using a large language model to perform iterative implicit feedback enhancement processing on text information includes: Using the text information and the information of the project to be recommended, a first prompt instruction is generated and input into the large language model to obtain positive and negative sample data selected by the model, which constitutes sample training data. The sample training data and the pre-acquired original training data are merged into recommendation data; and the recommendation system is iteratively trained using the recommendation data to obtain enhanced text information after training is completed.

2. The method according to claim 1, characterized in that, The process of using a large language model to perform contrastive semantic enhancement on text information includes: item attribute enhancement and user profile enhancement based on contrastive semantic enhancement. Using the text information, a second prompt instruction is generated, input into the large language model, and auxiliary information output by the model is obtained; and the auxiliary information is used to generate preliminarily enhanced text information. The text information and the pre-enhanced text information are respectively input into a pre-trained language model for encoding to generate a first encoding vector corresponding to the text information and a second encoding vector corresponding to the pre-enhanced text information. The first and second coding vectors are normalized to obtain the normalized first coding vector and the normalized second coding vector. Contrastive learning training is performed using a loss function based on noise contrast estimation to narrow the distance between the normalized first encoding vector and the normalized second encoding vector until a preset condition is met. At this point, training stops, and the enhanced text information that has completed the contrastive learning training is obtained.

3. The method according to claim 1, characterized in that, The generation of the text feature vector corresponding to the enhanced text information, the image feature vector corresponding to the image information, and the video feature vector corresponding to the video information includes: The enhanced text information is encoded using the BGE-m3 model to obtain text feature vectors; and the image information and video information are encoded using the GME model to obtain image feature vectors and video feature vectors, respectively.

4. The method according to claim 1, characterized in that, The image-text comparison learning strategy optimizes the semantic alignment of the text feature vectors, image feature vectors, and video feature vectors, including: Calculate the similarity between image feature vectors and text feature vectors, and the similarity between text feature vectors and video feature vectors; and use the obtained similarity to calculate the value of the view-text comparison loss function until the value meets the preset condition, then stop the semantic alignment optimization processing of the text feature vectors, image feature vectors, and video feature vectors, to obtain the optimized text feature vectors, optimized image feature vectors, and optimized video feature vectors.

5. The method according to claim 1, characterized in that, The process of fusing the optimized text feature vector, optimized image feature vector, and optimized video feature vector to obtain a multimodal unified feature vector includes: Through a cross-attention mechanism, the optimized text feature vector, optimized image feature vector, and optimized video feature vector are input into the W-layer Transformer structure for deep interaction and fusion layer by layer to obtain a multimodal unified feature vector; where W is a positive integer greater than 0.

6. The method according to any one of claims 1-5, characterized in that, The recommendation model is based on the group relative policy optimization GRPO method, and is obtained by training a large language model in two stages: supervised fine-tuning and reinforcement learning, using supervised fine-tuning datasets and reinforcement learning datasets.

7. A recommendation device based on a large language model, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-6.

9. A computer program product, characterized in that, The program product includes a computer program that, when executed by an electronic product, can implement the method described in any one of claims 1-6.

Images