Multi-modal content collaborative recommendation method based on large model

Abstract

The invention discloses a multi-modal content collaborative recommendation method based on a large model. The method comprises the steps that input is constructed, text information adopts a title of a to-be-recommended article, visual information obtains a feature vector of an article picture through a visual coding module, and collaborative information obtains hidden space vector representation of a user and the article through a collaborative coding module; information coding: integrating the recommendation data according to a template of the prompt information, coding the recommendation data and inputting the recommendation data into an LLM (Logical Link Model) to generate recommendation; and multi-stage fine tuning: enabling the model to master the input multi-modal information through three-stage fine tuning in combination with a task adapter. According to the method, differentiated interests and tendencies embodied when a user faces different modal information are emphasized, multi-modal information can be integrated, text information, visual information and collaborative information are uniformly incorporated into a large model, a task adapter module is introduced for the visual information and the collaborative information, and the user experience is improved. The modal characteristics highly related to the specific recommendation task can be extracted, so that the recommendation performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence recommendation algorithms, and specifically to a multimodal content collaborative recommendation method based on a large model. Background Art

[0002] In today's digital society, users are faced with an extremely large amount of online content and services. The recommendation system intelligently analyzes user behavior, interests, and historical records, filters and presents the most relevant content, effectively reducing the pressure on users in the selection process and improving the efficiency of information acquisition.

[0003] The current mainstream recommendation algorithms include: 1) collaborative filtering-based recommendation algorithms, which analyze users' historical behavior data to identify similarities between users or items, thereby achieving personalized recommendations; 2) content-based recommendation algorithms. Content-based recommendation algorithms start from the characteristics of the items themselves and infer users' possible interests and preferences by analyzing the content features of the items (such as text descriptions, visual information, and other attribute features).

[0004] With the continuous development of large language model technology, the research on recommendation algorithms based on large models has gradually attracted the research interest of the academic community in recent years. Although some progress has been made in the recommendation system based on large models, there are still some limitations. For example, collaborative filtering information has been proven to play an important role in the recommendation system, but most of the recommendation work based on large models ignores the collaborative interaction information between users and items, which may result in suboptimal recommendation effects. At the same time, user decisions in the real world are often the result of a combination of multiple factors, and single-modal large model recommendations are difficult to fully reflect this complexity. For example, in an e-commerce environment, the reason why a user clicks to buy a product may be because the detailed text description of the product meets their needs, or it may be because of the emotional resonance or visual appeal caused by the product picture display. Therefore, relying solely on the learning and reasoning of a single type of information by a large model is not enough to provide the best recommendation. Building and improving a multimodal recommendation system that integrates collaborative filtering mechanisms will be an important direction for the development of recommendation technology in the future. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a multimodal content collaborative recommendation method based on a large model, emphasizing the differentiated interest tendencies of users when faced with different modal information, and proposes a new method for integrating multimodal information, that is, incorporating text information, visual information, and collaborative information into a large model. Secondly, a task adapter module is introduced for visual information and collaborative information, aiming to extract modal characteristics that are highly relevant to specific recommendation tasks, thereby improving recommendation performance.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] The present invention is a multimodal content collaborative recommendation method based on a large model, comprising the following operations:

[0008] Construct input, where the text information uses the title of the item to be recommended, the visual information uses the visual encoding module to obtain the feature vector of the item image, and the collaborative information uses the collaborative encoding module to obtain the latent space vector representation of the user and the item;

[0009] Information encoding, which integrates the recommendation data according to the template of prompt information, and then encodes it and inputs it into LLM to generate recommendations;

[0010] Multi-stage fine-tuning: Through three-stage fine-tuning and in conjunction with the task adapter, the model can better grasp the input multimodal information, thereby improving the recommendation performance.

[0011] A further improvement of the present invention is that: the construction input uses a prefabricated prompt template as input information, the template is as follows: Figure 2 As shown, <historical item_title> represents a series of item titles that the user has browsed historically, and <target item_title> represents the title of the item to be recommended. In order to utilize visual information and collaborative information, this method introduces <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields. These fields do not contain any semantics and only serve as placeholders to prepare for the subsequent injection of latent feature vectors.

[0012] A further improvement of the present invention is that: for the information encoding, for all text content in the prompt information, the method uses the built-in tagging and embedding mechanism of the large model to convert it into tag embedding, and when processing other modal content, including <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields, the task adapter-based visual encoding module and the task adapter-based collaborative encoding module are respectively used to provide modal information for the large model;

[0013] In order to obtain the visual information of the object, this method uses a task adapter-based visual encoding module to map the image into a latent space vector:

[0014]

[0015] Among them, I i represents the image of the i-th item that the user has clicked before, N is the number of historical interaction records, and α i is the learnable weight parameter, f θv is the image feature extractor, θ v Represents the parameters of the image feature extractor, Adapter for image-recommendation tasks, is the parameter of the adapter, f up and f down Represent the dimensionality increase and decrease mapping layers respectively, σ is a nonlinear activation layer, I is the image of the item to be recommended, V_history is the visual latent space vector that is finally input to the big model to replace <historical item_image>, this vector contains the key feature information of the historical interaction item images, V_target is the visual latent space vector that is finally input to the big model to replace <target item_image>, this vector contains the key feature information of the item image to be recommended;

[0016] In order to obtain the collaborative information of items and users, this method uses a collaborative encoding module based on a task adapter to obtain the latent space vector representation of users and items:

[0017]

[0018] Among them, u and i represent users and items respectively, u and i represent the collaborative information representation of users and items respectively, and f θc is the process of obtaining this representation using the collaborative filtering model, θ c Represents the collaborative filtering model parameters, Recommend task adapter for collaboration, The parameters for this adapter.

[0019] A further improvement of the present invention is that the multi-stage fine-tuning adopts a three-stage fine-tuning strategy, wherein the three stages all use input information to generate prediction results and minimize the probability of prediction errors to achieve the training purpose. In the first stage, the method selects Figure 2 In the prompt template 1, no visual information and collaborative information are introduced in this stage, and only the LoRA module of the large model is fine-tuned to make the large model have a certain recommendation ability; in the second stage, this method selects Figure 2 In this stage, visual information is introduced and the large model is frozen. Only the image-recommendation task adapter and attention weight coefficient are trained, so that the model can correctly use visual information to improve the recommendation performance. In the third stage, this method selects Figure 2 In the prompt template 3, text information, visual information, and collaborative information are all introduced at this stage, and the large model and image-recommendation task adapter are frozen, and only the collaborative-recommendation task adapter is trained, so that the model finally has the recommendation ability to combine multimodal information and integrate collaborative filtering mechanism.

[0020] The beneficial effects of the present invention are as follows: the present invention proposes a multimodal content collaborative recommendation method based on a large model, by constructing an input, wherein the text information adopts the title of the item to be recommended, the visual information obtains the feature vector of the item image through a visual encoding module, and the collaborative information obtains the latent space vector representation of the user and the item by utilizing the collaborative encoding module; information encoding, integrating the recommendation data according to the template of the prompt information, and then encoding it and inputting it into the LLM to generate a recommendation; multi-stage fine-tuning, through three-stage fine-tuning and cooperating with a task adapter, the model can better grasp the input multimodal information, thereby improving the recommendation performance, emphasizing the differentiated interest tendencies of users when facing different modal information, and proposing a new method for integrating multimodal information, that is, integrating text information, visual information, and collaborative information into a large model; secondly, a task adapter module is introduced for visual information and collaborative information, aiming to extract modal characteristics that are highly relevant to specific recommendation tasks, thereby improving the recommendation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a large model-based multimodal content collaborative recommendation algorithm framework provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of prompt templates corresponding to different training stages provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] See also Figure 1 , a multimodal content collaborative recommendation method based on a large model in this embodiment includes:

[0025] Construct input, where the text information uses the title of the item to be recommended, the visual information uses the visual encoding module to obtain the feature vector of the item image, and the collaborative information uses the collaborative encoding module to obtain the latent space vector representation of the user and the item;

[0026] Information encoding, which integrates the recommendation data according to the template of prompt information, and then encodes it and inputs it into LLM to generate recommendations;

[0027] Multi-stage fine-tuning: Through three-stage fine-tuning and in conjunction with the task adapter, the model can better grasp the input multimodal information, thereby improving the recommendation performance.

[0028] The construction input uses a pre-made prompt template as input information. The template is as follows: Figure 2 As shown, <historical item_title> represents a series of item titles that the user has browsed historically, and <target item_title> represents the title of the item to be recommended. In order to utilize visual information and collaborative information, this method introduces <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields. These fields do not contain any semantics and only serve as placeholders to prepare for the subsequent injection of latent feature vectors.

[0029] The information encoding method uses the built-in tagging and embedding mechanism of the large model to convert all text content in the prompt information into tag embedding. When processing other modal content, including <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields, the task adapter-based visual encoding module and the task adapter-based collaborative encoding module are used to provide modal information for the large model.

[0030] In order to obtain the visual information of the object, this method uses a task adapter-based visual encoding module to map the image into a latent space vector:

[0031]

[0032] Among them, I i represents the image of the i-th item that the user has clicked before, N is the number of historical interaction records, and α i is the learnable weight parameter, f θv is the image feature extractor, θ v Represents the parameters of the image feature extractor, Adapter for image-recommendation tasks, is the parameter of the adapter, f up and f down Represent the dimensionality increase and decrease mapping layers respectively, o is the nonlinear activation layer, I is the image of the item to be recommended, V_history is the visual latent space vector that is finally input to the big model to replace <historical item_image>, this vector contains the key feature information of the historical interaction item image, V_target is the visual latent space vector that is finally input to the big model to replace <target item_image>, this vector contains the key feature information of the item image to be recommended;

[0033] In order to obtain the collaborative information of items and users, this method uses a collaborative encoding module based on a task adapter to obtain the latent space vector representation of users and items:

[0034]

[0035] Among them, u and i represent users and items respectively, u and i represent the collaborative information representation of users and items respectively, and f θc is the process of obtaining this representation using the collaborative filtering model, θ c Represents the collaborative filtering model parameters, Recommend task adapter for collaboration, The parameters for this adapter.

[0036] The multi-stage fine-tuning adopts a three-stage fine-tuning strategy, in which all three stages use input information to generate prediction results and minimize the probability of prediction errors to achieve the training purpose. In the first stage, this method selects Figure 2 In the prompt template 1, no visual information and collaborative information are introduced in this stage, and only the LoRA module of the large model is fine-tuned to make the large model have a certain recommendation ability; in the second stage, this method selects Figure 2 In this stage, visual information is introduced and the large model is frozen. Only the image-recommendation task adapter and attention weight coefficient are trained, so that the model can correctly use visual information to improve the recommendation performance. In the third stage, this method selects Figure 2 In the prompt template 3, text information, visual information, and collaborative information are all introduced at this stage, and the large model and image-recommendation task adapter are frozen, and only the collaborative-recommendation task adapter is trained, so that the model finally has the recommendation ability to combine multimodal information and integrate collaborative filtering mechanism.

[0037] Through the introduction of the above embodiments, the present invention adopts a multimodal content collaborative recommendation method based on a large model, by constructing an input, wherein the text information adopts the title of the item to be recommended, the visual information obtains the feature vector of the item image through the visual encoding module, and the collaborative information obtains the latent space vector representation of the user and the item by utilizing the collaborative encoding module; information encoding, integrating the recommendation data according to the template of the prompt information, and then encoding it and inputting it into the LLM to generate a recommendation; multi-stage fine-tuning, through three-stage fine-tuning and cooperating with the task adapter, the model can better grasp the input multimodal information, thereby improving the recommendation performance, emphasizing the differentiated interest tendencies of users when facing different modal information, and proposing a new method for integrating multimodal information, that is, integrating text information, visual information, and collaborative information into a large model, and secondly introducing a task adapter module for visual information and collaborative information, aiming to extract modal characteristics that are highly relevant to specific recommendation tasks, thereby improving the recommendation performance.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined similarly here.

[0039] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal content collaborative recommendation method based on a large model, characterized in that: include: Construct input, where the text information uses the title of the item to be recommended, the visual information uses the visual encoding module to obtain the feature vector of the item image, and the collaborative information uses the collaborative encoding module to obtain the latent space vector representation of the user and the item; Information encoding, which integrates the recommendation data according to the template of prompt information, and then encodes it and inputs it into LLM to generate recommendations; Multi-stage fine-tuning: Through three-stage fine-tuning and in conjunction with the task adapter, the model can better grasp the input multimodal information, thereby improving the recommendation performance.

2. According to claim 1, a multimodal content collaborative recommendation method based on a large model is characterized in that: The construction input adopts a pre-made prompt template as input information, and the template is shown in Figure 2, wherein <historical item_title> represents a series of item titles that the user has browsed historically, and <target item_title> represents the title of the item to be recommended. In order to utilize visual information and collaborative information, this method introduces <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields. These fields do not contain any semantics and only serve as placeholders to prepare for the subsequent injection of latent feature vectors.

3. According to the large model-based multimodal content collaborative recommendation method of claim 1, it is characterized in that: The information encoding method uses the built-in tagging and embedding mechanism of the large model to convert all text content in the prompt information into tag embedding. When processing other modal content, including <historical item_image>, <target item_image>, <collaborative information_user>, and <collaborative information_item> fields, the task adapter-based visual encoding module and the task adapter-based collaborative encoding module are used to provide modal information for the large model. In order to obtain the visual information of the object, this method uses a task adapter-based visual encoding module to map the image into a latent space vector: Among them, I i represents the image of the i-th item that the user has clicked before, N is the number of historical interaction records, and α i is the learnable weight parameter, f θv is the image feature extractor, θv represents the parameters of the image feature extractor, Adapter for image-recommendation tasks, is the parameter of the adapter, f up and f down Represent the dimensionality increase and decrease mapping layers respectively, σ is a nonlinear activation layer, I is the image of the item to be recommended, V_history is the visual latent space vector that is finally input to the big model to replace <historical item_image>, this vector contains the key feature information of the historical interaction item images, V_target is the visual latent space vector that is finally input to the big model to replace <target item_image>, this vector contains the key feature information of the item image to be recommended; In order to obtain the collaborative information of items and users, this method uses a collaborative encoding module based on a task adapter to obtain the latent space vector representation of users and items: Among them, u and i represent users and items respectively, u and i represent the collaborative information representation of users and items respectively, and f θc is the process of obtaining this representation using the collaborative filtering model, θc represents the collaborative filtering model parameters, Recommend task adapter for collaboration, The parameters for this adapter.

4. According to claim 1, a multimodal content collaborative recommendation method based on a large model is characterized in that: The multi-stage fine-tuning adopts a three-stage fine-tuning strategy, wherein all three stages utilize input information to generate prediction results and minimize the probability of prediction errors to achieve the training purpose. In the first stage, the method selects prompt template 1 in FIG2 , and does not introduce visual information and collaborative information in this stage, and only fine-tunes the LoRA module of the large model, so that the large model has a certain recommendation capability; in the second stage, the method selects prompt template 2 in FIG2 , and introduces visual information in this stage, and freezes the large model, and only trains the image-recommendation task adapter and the attention weight coefficient, so that the model can correctly use visual information to improve the recommendation performance; in the third stage, the method selects prompt template 3 in FIG2 , and introduces text information, visual information, and collaborative information in this stage, and freezes the large model and the image-recommendation task adapter, and only trains the collaborative-recommendation task adapter, so that the model finally has the recommendation capability of combining multimodal information and integrating the collaborative filtering mechanism.

Images