Multi-modal topic tag recommendation method based on large language model

Abstract

The invention provides a multi-modal topic tag recommendation method based on a large language model, and the method comprises the steps: obtaining to-be-recommended information and historical recommendation information, inputting a to-be-recommended image and a historical image into a pre-trained image encoder, and carrying out the encoding processing, a to-be-recommended image code and a historical image code are obtained through projection layer neural network mapping; then inputting into a sequence-independent extractor to generate a soft prompt mark; and fusing the soft prompt mark with a pre-constructed hard prompt mark to obtain a mixed mark, inputting the mixed mark, the to-be-recommended text and the historical text into a pre-trained large language model for reasoning, and finally determining a topic tag recommendation result of the to-be-recommended information according to hidden state features obtained by reasoning. By means of the topic label recommendation method and device, the defects that in the prior art, when a large language model and multi-modal learning are combined for topic label recommendation, the interest requirements of different users cannot be met, and the recommendation accuracy and efficiency are low are overcome.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal topic tag recommendation method based on a large language model. Background Art

[0002] Social media platforms have become an indispensable part of people's daily lives. Users share information and express their opinions by posting multimodal content such as text, pictures, and videos. In order to better organize and retrieve this massive amount of information, hashtags are widely used. Hashtags not only help users quickly find content of interest, but also provide the platform with valuable user behavior and interest data.

[0003] However, in the field of topic tag recommendation, manually selecting appropriate topic tags for each post is often time-consuming and inaccurate, so automated topic tag recommendation systems are becoming increasingly important. Traditional tag recommendation methods are mainly based on text analysis, such as TF-IDF, topic models, etc., but these methods are difficult to fully utilize multimodal information such as images. Although the prior art proposes to combine large language models with multimodal learning for topic tag recommendation, this can not only utilize the powerful text understanding and generation capabilities of large language models, but also effectively process multimodal information such as images, providing a new solution for topic tag recommendation. However, large language models usually require a lot of computing resources, and the feature representation and distribution of different modal information data are often different, and the interests and needs of different users are also different, resulting in low accuracy and efficiency of the final topic tag recommendation. Summary of the invention

[0004] The present invention provides a multimodal topic tag recommendation method based on a large language model, so as to overcome the defects in the prior art that when a large language model is combined with multimodal learning to perform topic tag recommendation, the interest requirements of different users cannot be met and the recommendation accuracy and efficiency are low.

[0005] The present invention provides a multimodal topic tag recommendation method based on a large language model, the method comprising the following steps: Acquire information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; Inputting the image to be recommended and the historical image into a pre-trained image encoder for encoding, and performing feature mapping on the image features obtained by the encoding through a projection layer neural network to obtain corresponding encodings of the image to be recommended and historical images; Inputting the to-be-recommended image code and the historical image code into an order-independent extractor for generation to obtain a soft prompt mark; Pre-constructing a hard prompt tag for the information to be recommended, fusing the soft prompt tag with the hard prompt tag to obtain a mixed tag, and inputting the mixed tag, the text to be recommended, and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; The hidden state features are mapped through a fully connected layer neural network to obtain a probability distribution of topic tags for the information to be recommended, and the topic tag with the largest probability in the topic tag probability distribution is used as a topic tag recommendation result for the information to be recommended.

[0006] In some embodiments, the order-independent extractor includes a routing module, a cross-fusion module, and a self-fusion module, and the step of inputting the image code to be recommended and the historical image code into the order-independent extractor to generate a soft prompt mark includes: Calling the self-fusion module to perform self-attention mechanism processing on the encoding of the image to be recommended to obtain self-fusion image features; Calling the cross-fusion module to perform cross-attention mechanism processing on the self-fusion image feature and the historical image encoding to obtain a cross-fusion image feature; Calling the routing module to perform routing calculation on the image code to be recommended and the historical image code to obtain a relative attention weight; The cross-fusion image features are weightedly summed by the relative attention weights, and the image features obtained by the weighted sum are fused with the self-fusion image features to obtain soft prompt marks.

[0007] In some embodiments, the self-attention mechanism processing is performed on the encoding of the image to be recommended by the self-fusion module to obtain the self-fusion image features, including: Obtaining a learnable soft prompt for the information to be recommended; splicing the learnable soft prompt with the image code to be recommended to obtain a spliced ​​image feature; The self-fusion module performs self-attention calculation on the spliced ​​image features to obtain self-fusion image features.

[0008] In some embodiments, calling the cross-fusion module to perform cross-attention mechanism processing on the self-fusion image features and the historical image encoding to obtain the cross-fusion image features includes: The historical image encoding is used as a key vector and a value vector of cross attention respectively; Using the self-fused image features as a query vector for cross attention; The cross-fusion module performs cross-attention calculation on the key vector, the value vector and the query vector to obtain a cross-fusion image feature.

[0009] In some embodiments, calling the routing module to perform routing calculation on the to-be-recommended image encoding and the historical image encoding to obtain a relative attention weight includes: Decomposing the image code to be recommended and the historical image code into a plurality of local representation tags respectively by a decomposer in the routing module; The local representation mark is processed by a cross-attention mechanism through a multi-head cross-attention layer in the routing module to obtain a relative attention weight.

[0010] In some embodiments, the number of the local representation tags is the same as the number of attention heads in the multi-head cross attention layer, and the multi-head cross attention layer in the routing module performs a cross attention mechanism on the local representation tags to obtain a relative attention weight, including: For each first local representation tag of the image to be recommended, mapping the first local representation tag to a cross-attention query feature; Connecting the plurality of second local representation tags encoded by the historical image into a historical tag sequence, and mapping the historical tag sequence into a key feature of cross attention; Performing cross-attention calculation on the query feature and the key feature through the attention head in the multi-head cross-attention layer, and obtaining a head attention weight of the attention head for the first local representation mark; The attention weights calculated for each attention head are used to calculate the mean attention value and obtain the relative attention weight.

[0011] In some embodiments, the topic tag probability distribution of the information to be recommended is predicted by calling a topic tag recommendation model, the topic tag recommendation model includes a pre-trained image encoder, a projection layer network, an order-independent extraction model, a pre-trained large language model, and a fully connected layer network. The training process of the topic tag recommendation model includes: Obtaining a sample of information to be recommended and a sample of historical recommended information, wherein the sample of information to be recommended carries a real recommendation tag corresponding to the topic tag; Inputting the image of the information sample to be recommended and the image of the historical recommended information sample into the pre-trained image encoder and the projection layer network to obtain information sample image features and historical sample image features; Inputting the information sample image features and the historical sample image features into the order-independent extraction model for generation to obtain a predicted soft label; The predicted soft label is fused with the pre-constructed real hard label to obtain a predicted mixed label, and the predicted mixed label, the text of the information sample to be recommended, and the text of the historical recommended information sample are input into the pre-trained large language model for reasoning to obtain a predicted hidden state feature; Mapping the predicted hidden state features through the fully connected layer network to obtain predicted recommendation labels, and constructing a cross entropy loss function based on the predicted recommendation labels and the true recommendation labels; Back propagation is performed in the topic tag recommendation model through the cross entropy loss function to update the parameters of the projection layer network, the order-independent extraction model, and the fully connected layer network.

[0012] The present invention also provides a multimodal topic tag recommendation device based on a large language model, the device comprising the following modules: An acquisition module, used to acquire information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; An encoding module, used for inputting the image to be recommended and the historical image into a pre-trained image encoder for encoding, and performing feature mapping on the image features obtained by the encoding through a projection layer neural network to obtain corresponding encodings of the image to be recommended and historical image; A generating module, used for inputting the image code to be recommended and the historical image code into an order-independent extractor for generating soft prompt marks; An inference module is used to pre-construct a hard prompt tag for the information to be recommended, fuse the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; The recommendation module is used to map the hidden state features through a fully connected layer neural network to obtain a probability distribution of topic tags of the information to be recommended, and use the topic tag with the highest probability in the probability distribution of topic tags as the topic tag recommendation result of the information to be recommended.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multimodal topic tag recommendation method based on a large language model as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal topic tag recommendation methods based on a large language model as described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal topic tag recommendation methods based on a large language model as described above.

[0016] The multimodal topic tag recommendation method based on a large language model provided by the present invention combines the information to be recommended and the historical recommendation information to recommend topic tags, and constructs soft prompt tags through image features, and then combines hard tags and text input into the large language model for reasoning, thereby realizing the recommendation of topic tags through the fusion prediction of multimodal information, which can not only meet the interests and needs of different users, but also improve the recommendation accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the multimodal topic tag recommendation method based on a large language model provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the principle of the multimodal topic tag recommendation method based on a large language model provided by the present invention.

[0020] Figure 3 It is a processing schematic diagram of the order-independent extractor provided by the present invention.

[0021] Figure 4 It is a processing schematic diagram of the routing module provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of a multimodal topic tag recommendation device based on a large language model provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The following describes the multimodal topic tag recommendation method based on a large language model of the present invention in conjunction with the accompanying drawings. Figure 1 is a flow chart of a multimodal topic tag recommendation method based on a large language model provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 105.

[0026] Step 101: Obtain information to be recommended and historical recommendation information.

[0027] In the application scenario where topic tag recommendation is required, the information to be recommended and the historical recommendation information need to be obtained first. The application scenario can be social media platforms such as forums, websites, and forums. The corresponding information can be posts, online comments, forum content, etc., including pictures, text, video or audio, etc. The information to be recommended can be information posted by users and topic tags need to be selected, including images to be recommended and text to be recommended for topic tag recommendation, while the historical recommendation information is the information record with topic tags that the user has sent in the past. There can be multiple, specifically including historical images and historical texts.

[0028] The embodiment of the present invention is described below using posts in a Tieba scenario as an example, so in the Tieba scenario, the post that the user currently needs to send and the user's historical posting record are obtained, including the post image, post text, historical posting record text and historical posting record image.

[0029] Step 102: input the image to be recommended and the historical image into a pre-trained image encoder for encoding, and perform feature mapping on the image features obtained by the encoding through a projection layer neural network to obtain corresponding image encoding to be recommended and historical image encoding.

[0030] After obtaining the information to be recommended and the historical recommended information, the images to be recommended and the historical images are input into the pre-trained image encoder for encoding. The pre-trained image encoder can be a (VisionTransformer, VIT) model, specifically a ViT-Base-Patch16-224 model, which is characterized by fixing the input image size to 224*224 pixels, and then dividing the input image into 16*16 pixel patches for linear mapping. Here, only the images in the information to be recommended and the historical recommended information are embedded to extract image features. Therefore, the pre-trained image encoder is called to extract features from the images to be recommended and the historical images respectively, and the corresponding image features are obtained through encoding.

[0031] In order to match the input feature dimension of the subsequent pre-trained large language model, the encoded image features are input into a projection layer neural network, and the image features obtained by encoding are feature mapped through the projection layer neural network to obtain the corresponding recommended image encoding and historical image encoding. The goal of feature mapping is mainly to adjust the feature dimension to be consistent with the input feature dimension of the large language model. The projection layer neural network can be a fully connected layer neural network.

[0032] For example Figure 2 As shown in the figure, after collecting data, stage 1 is performed, the post images and historical post record images for which the user needs to recommend topic tags are input into the image encoder for encoding, and feature mapping is performed after encoding to obtain the corresponding post image encoding and historical post record image encoding. This is then input into the order-independent extractor module to generate the corresponding soft prompt tags.

[0033] Step 103: Input the image code to be recommended and the historical image code into the order-independent extractor for generation to obtain a soft prompt mark.

[0034] After extracting the image features in step 102, the image code to be recommended and the historical image code are input into the order-independent extractor to generate soft prompt tags. The order-independent extractor is mainly used to eliminate the influence of the input order of image features on the large language model, and the soft prompt tag can be regarded as a label feature to help the large language model perform reasoning later. The generation process of the order-independent extractor is described in detail below.

[0035] In the embodiment of the present invention, the order-independent extractor includes three partial modules: a routing module, a cross-fusion module, and a self-fusion module. Figure 3As shown, the three modules of the order-independent extractor perform different processing on the input post image encoding and the historical post record image encoding respectively, including routing calculation of the routing module, cross-fusion processing of the cross-fusion module, and self-fusion processing of the self-fusion module. Finally, the processing results of the three modules are combined to obtain the soft prompt output of the order-independent extractor, which are explained one by one below.

[0036] First, the self-fusion module is called to process the recommended image encoding with the self-attention mechanism to obtain the self-fusion image features. Figure 3 As shown in Figure 1, the processing process of the self-fusion module is to concatenate the post image encoding (i.e., the image encoding to be recommended) and the input soft label, and then perform self-fusion. Here, the self-fusion module receives the input learnable soft hint in order to enable the order-independent extractor to pay more attention to the important areas and features inside the image to be recommended.

[0037] Specifically, learnable soft hints are obtained for the information to be recommended. The learnable soft hints here can be regarded as label data of the information to be recommended, which can be constructed by learning the semantic model or constructed manually. Here, the learnable soft hints are regarded as a learnable initial parameter to help the self-fusion module perform self-attention calculation, denoted as . Then the learnable soft hint and the image code to be recommended (denoted as ) to obtain the spliced ​​image features, and the self-fusion module performs self-attention calculation on the spliced ​​image features to obtain the self-fusion image features, denoted as , the specific calculation can be expressed as the following formula (1) and formula (2): (1) (2) In the above formula (1) and formula (2), Indicates the image encoding to be recommended, represents a learnable soft tag, concat represents a feature concatenation function, , , They represent the query vector, key vector, and value vector of the self-attention calculation respectively. Represents the self-attention calculation function.

[0038] Next, the cross-fusion module is called to perform cross-attention processing on the self-fusion image features and the historical image encoding to obtain the cross-fusion image features. Figure 3As shown, the processing process of the self-fusion module is to cross-fuse the self-fusion image features output by the self-fusion module with the image encoding of the historical posting record (i.e., the historical image features) in order to cross-fuse the historical image features with the features of the image to be recommended, so that the order-independent extractor can capture the association relationship between different images, that is, the association relationship between the historical image and the image to be recommended.

[0039] Specifically, the historical image encoding is first used as the key vector and value vector of the cross attention, and the self-fused image feature is used as the query vector of the cross attention. Finally, the cross-fusion module performs cross-attention calculation on the key vector, value vector and query vector to obtain the cross-fusion image feature, which is recorded as , the calculation process can be expressed as the following formula (3) to formula (5): (3) (4) (5) In the above formulas (3) to (5), represents the self-fusion image features, represents the query vector for the cross-attention computation, , denote the key vector and value vector of the query vector for cross-attention calculation, respectively. Represents historical image coding, represents the code of the nth historical image, n represents the total number of historical images, Represents the cross attention calculation function.

[0040] Then the routing module is called to perform routing calculations on the recommended image encoding and the historical image encoding to obtain the relative attention weights. Figure 3 As shown, the routing module inputs the historical post record image code and the post image code, and the routing control module calculates the specific correlation between the image to be recommended and the historical image, and calculates the corresponding attention weight.

[0041] Specifically, the decomposer in the routing module first decomposes the image code to be recommended and the historical image code into multiple local representation tags. Both the image code to be recommended and the historical image code carry the [CLS] tag, which is used to represent the global features of the entire image. Therefore, the [CLS] tag can be directly decomposed here. The number of local representation tags is denoted as m, and the value can be determined according to the specific situation. For example, if m is 4, it is decomposed into 4 tags. The decomposer can also be a fully connected layer neural network, which is used to map the global [CLS] tag into multiple local representation tags.

[0042] After the tag decomposition is completed, the local representation tags are processed by the multi-head cross-attention layer in the routing module through the cross-attention mechanism to obtain the relative attention weight. Here, there are multiple attention heads in the multi-head cross-attention layer, and each attention head processes a local representation tag to calculate the attention of the local features of the image. Therefore, the number of local representation tags is the same as the number of attention heads in the multi-head cross-attention layer. If the number of local representation tags m is 4, then the number of attention heads is also 4.

[0043] During the cross-attention mechanism processing, the recommended image is encoded Each first partial representation of a label is denoted by , indicating the image encoding to be recommended The i-th first local representation tag maps the first local representation tag to the query feature of the cross attention, and encodes the historical image Multiple second local representation marks of , indicating historical image coding The i-th second local representation mark , then, connect them into a historical tag sequence, denoted as , and maps the historical tag sequence into the key features of the cross-attention. Through the attention heads in the multi-head cross-attention layer (denoted as , i represents the i-th attention head) performs cross attention calculation on the query feature and the key feature to obtain the head attention weight of the attention head for the first local representation mark, and finally calculates the attention mean of the head attention weights calculated for each attention head to obtain the relative attention weight, recorded as , expressed as the following formula (6): (6) In the above formula (6), represents the cross-attention calculation function of the i-th attention head in the multi-head cross-attention layer, Indicates the image encoding to be recommended The i-th first local representation mark, Represents historical image coding The i-th second local representation mark , m represents the number of local representation tags, or the number of attention heads in the multi-head cross attention layer, and n represents the number of historical image encodings, that is, the number of historical images.

[0044] For example, Figure 4As shown, in the routing module, the historical posting record image code and the post image code are input into the decomposer for decomposition, and 4 local tags are obtained respectively (namely, the first local tag and the second local tag mentioned above), and then the multi-head cross-attention calculation is performed, wherein the multi-head cross-attention layer includes 4 attention heads, and each attention head is calculated to obtain an attention weight (namely, the head attention weight mentioned above), and finally the mean sum operation is performed to obtain the corresponding attention weight.

[0045] In this way, the processing process of the three modules of the order-independent extractor is introduced. However, it should be noted that the processing processes of the routing module, the cross-fusion module and the self-fusion module are carried out simultaneously, that is, there is no distinction between the order of precedence. For the convenience of explanation, the embodiment of the present invention has a distinction between the order of precedence, but this should not be limited in actual implementation.

[0046] After the three modules of the order-independent extractor are processed, the relative attention weights Cross-fusion image features Perform weighted summation and combine the image features obtained by weighted summation with the self-fusion image features After fusion processing, a soft prompt mark is obtained, which is recorded as , the calculation process is expressed as the following formula (6) to formula (9): (6) (7) (8) (9) In the above formula (6), represents the cross-fusion image features, represents the cross-fusion features of the historical image encoding of the nth historical image and the image encoding to be recommended, and W is the attention weight calculated by the routing module , That is, it represents the relative attention weight calculated by the routing module for the historical image encoding of the nth historical image. Represents self-fused image features.

[0047] For example, Figure 3 As shown in the figure, after the routing module calculates the attention weight, the cross-fusion image features are weighted and summed by the attention weight, that is, the weighted sum is added, and finally the weighted summation result is further fused with the self-fusion image features output by the self-fusion module to finally obtain the corresponding soft prompt output.

[0048] The order-independent extractor generates a soft prompt tag for the information to be recommended, which can be a prompt text or a tag feature. It can be used for reasoning of subsequent large language models or as training data to help train the order-independent extractor.

[0049] In the embodiment of the present invention, the three modules of the order-independent extractor are used to generate the input image encoding to be recommended and the historical image encoding respectively, and finally output the corresponding soft prompt output, so as to eliminate the influence of the input order on the subsequent large language model prediction and improve the prediction accuracy of the large language model.

[0050] Step 104: pre-construct a hard prompt tag for the information to be recommended, fuse the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended, and the historical text into a pre-trained large language model for inference to obtain hidden state features.

[0051] After the soft hint tags are obtained through the order-independent extractor, the next step is the inference process of the pre-trained large language model. Figure 2 As shown, after the order-independent extractor module outputs the soft label tag, it enters the reasoning process of the large language model in stage 2.

[0052] In order to improve the inference accuracy of the large language model, the embodiment of the present invention pre-constructs a hard prompt tag for the information to be recommended (i.e. Figure 2 The hard prompt shown in stage 2 in the figure is manually constructed and is mainly used to prompt the large language model for reasoning. The example template can be "provide topic tags for this post". The hard prompt provides clear instructions for the reasoning task of the large language model.

[0053] Next, the soft prompt tag and the hard prompt tag are fused to obtain a mixed tag, and the mixed tag is directly fused here to form a mixed tag, that is, the image visual information of the information to be recommended is integrated into the prompt text. Finally, the mixed tag, the text to be recommended, and the historical text are directly input into the pre-trained large language model for reasoning to obtain hidden state features. Here, the text to be recommended in the information to be recommended, the historical text in the historical recommended information, and the text corresponding to the mixed tag are input into the large language model. The large language model uses internal knowledge and reasoning ability, takes the mixed tag as a prompt word, extracts and fuses the semantic features of the text to be recommended and the historical text, and finally generates hidden representation features related to the topic tag, that is, hidden state features.

[0054] For example, Figure 2As shown, after entering stage 2, the mixed tags, historical posting record text and post text are input into the large language model for reasoning to generate hidden representation features related to topic tags, namely hidden state features.

[0055] The embodiment of the present invention constructs soft prompt tags through image features, and then combines hard tags and text input into a large language model for reasoning, thereby realizing the recommendation of topic tags through the fusion prediction of multimodal information, thereby improving the accuracy and efficiency of topic tag recommendations.

[0056] Step 105: Map the hidden state features through a fully connected layer neural network to obtain a probability distribution of topic tags for the information to be recommended, and use the topic tag with the highest probability in the probability distribution of topic tags as a topic tag recommendation result for the information to be recommended.

[0057] Through step 104, after the large language model outputs the hidden state features, the hidden state features are mapped through the fully connected layer neural network to obtain the topic tag probability distribution of the information to be recommended, and the topic tag with the largest probability in the topic tag probability distribution is used as the topic tag recommendation result of the information to be recommended.

[0058] Here, a fully connected neural network is set up, and then the input hidden state features are feature mapped to obtain the probability distribution of topic tags for the information to be recommended. The categories of topic tags can be predefined according to the semantic content of the information to be recommended, such as historical categories, literary categories, geographical categories, etc. In this way, multiple topic tags that may be suitable for the information to be recommended can be predefined as candidate topic tags. The probability distribution of topic tags obtained by mapping the fully connected neural network layer includes the predicted probability corresponding to each candidate topic tag.

[0059] Furthermore, the topic tag with the largest probability in the topic tag probability distribution is used as the topic tag recommendation result of the information to be recommended.

[0060] like Figure 2 As shown in Figure 2, in stage 2, after the large language model outputs the hidden state features, the hidden state features are input into the fully connected layer to obtain the probability distribution of multiple predefined candidate tags, and then the candidate topic tag with the highest probability is selected from the probability distribution as the recommended tag for the user to send the current post. For example, in the probability distribution, the probability corresponding to the geographical category is the highest, so it can be determined that the recommended tag for the user to send the current post is the geographical tag.

[0061] In some embodiments, considering the multimodal topic tag recommendation method based on a large language model provided by the present invention, multiple network models are involved, such as an image encoder, a projection layer neural network, an order-independent extraction model, a large language model, and a fully connected layer network. During model training, it may be necessary to perform training or parameter fine-tuning separately, and it is also necessary to ensure the connection between models, which increases the difficulty of model training, affects the performance of the model, and also significantly increases the computational overhead of the model.

[0062] In order to reduce the computational overhead of model training, a topic tag recommendation model is proposed in an embodiment of the present invention. By calling the topic tag recommendation model, the topic tag probability distribution of the information to be recommended is predicted. The topic tag recommendation model includes a pre-trained image encoder, a projection layer network, an order-independent extraction model, a pre-trained large language model, and a fully connected layer network. These models are integrated into a topic tag recommendation model to achieve seamless connection between multiple models. The training process of the topic tag recommendation model is introduced below.

[0063] First, we obtain samples of information to be recommended and samples of historical recommended information, where the samples of information to be recommended carry real recommended labels corresponding to topic labels, and these samples are used to train the topic label recommendation model. The real recommended labels are the real topic labels corresponding to the samples of information to be recommended, and both the samples of information to be recommended and the samples of historical recommended information have images and texts.

[0064] Next, the image of the information sample to be recommended and the image of the historical recommended information sample are input into the pre-trained image encoder and the projection layer network to obtain the information sample image features and the historical sample image features, and then the information sample image features and the historical sample image features are input into the order-independent extraction model for generation to obtain the predicted soft label. In this process, soft label prompts generated by artificial construction or other models can also be input to help the order-independent extraction model generate predicted soft labels.

[0065] The predicted soft label is fused with the pre-built real hard label to obtain the predicted mixed label, and the predicted mixed label, the text of the information sample to be recommended, and the text of the historical recommended information sample are input into the pre-trained large language model for inference to obtain the predicted hidden state features. The predicted hidden state features are mapped through the fully connected layer network to obtain the predicted recommended label, thus completing the introduction of the forward propagation process of the topic label recommendation model.

[0066] Finally, based on the predicted recommended tags and the real recommended tags, a cross entropy loss function is constructed, that is, the difference between the topic tags predicted by the topic tag recommendation model and the real tags is compared and represented by the cross entropy loss function. Then, the cross entropy loss function is used to perform back propagation in the topic tag recommendation model to update the parameters of the projection layer network, the order-independent extraction model, and the fully connected layer network.

[0067] Here, during the training process, the model gradient is calculated through the back propagation of the cross entropy loss function, and then the gradient is optimized through the gradient optimization algorithm to update the parameters of the topic tag recommendation model. However, when updating the parameters, the parameters of the pre-trained image encoder and the pre-trained large language model are frozen and do not need to be updated. Therefore, during the back propagation of the loss function, only the parameters of the projection layer network, the order-independent extraction model, and the fully connected layer network are updated.

[0068] In the embodiment of the present invention, a separate topic tag recommendation model is designed and trained to predict the probability distribution of topic tags for recommended information. During the training process, the parameters of a part of the modules are frozen, and only the parameters of another part of the model are trained. In this way, the training difficulty of the model can be effectively reduced, the performance of the model can be guaranteed, and the computational overhead of the model can be significantly reduced.

[0069] The multimodal topic tag recommendation device based on a large language model provided by the present invention is described below. The multimodal topic tag recommendation device based on a large language model described below and the multimodal topic tag recommendation method based on a large language model described above can refer to each other.

[0070] like Figure 5As shown, the multimodal topic tag recommendation device based on the large language model includes: an acquisition module 501, an encoding module 502, a generation module 503, an inference module 504, and a recommendation module 505. Specifically, the acquisition module 501 is used to obtain information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; the encoding module 502 is used to input the images to be recommended and the historical images into a pre-trained image encoder for encoding processing, and perform feature mapping on the image features obtained by the encoding processing through a projection layer neural network to obtain the corresponding image encoding to be recommended and the historical image encoding; the generation module 503 is used to input the image encoding to be recommended and the historical image encoding into an order-independent image encoder. The inference module 504 is used to pre-build a hard prompt tag for the information to be recommended, merge the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; the recommendation module 505 is used to map the hidden state feature through a fully connected layer neural network to obtain a topic tag probability distribution of the information to be recommended, and use the topic tag with the largest probability in the topic tag probability distribution as the topic tag recommendation result of the information to be recommended.

[0071] It should be noted that the beneficial effects of the multimodal topic tag recommendation device based on the large language model here and the multimodal topic tag recommendation method based on the large language model mentioned above can correspond to each other, so the beneficial effects of the multimodal topic tag recommendation device based on the large language model will not be repeated here.

[0072] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a multimodal topic tag recommendation method based on a large language model, the method comprising: obtaining information to be recommended and historical recommendation information, wherein the information to be recommended includes an image to be recommended and a text to be recommended for topic tag recommendation, and the historical recommendation information includes a historical image and a historical text; inputting the image to be recommended and the historical image into a pre-trained image encoder for encoding processing, and performing feature mapping on the image features obtained by the encoding processing through a projection layer neural network to obtain corresponding image encoding to be recommended and historical image encoding; inputting the image encoding to be recommended and the historical image encoding into an order-independent extractor for generation to obtain a soft prompt tag; pre-constructing a hard prompt tag for the information to be recommended, fusing the soft prompt tag with the hard prompt tag to obtain a mixed tag, and inputting the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; mapping the hidden state feature through a fully connected layer neural network to obtain a topic tag probability distribution of the information to be recommended, and taking the topic tag with the largest probability in the topic tag probability distribution as the topic tag recommendation result of the information to be recommended.

[0073] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal topic tag recommendation method based on a large language model provided by the above methods, the method comprising: obtaining information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; inputting the images to be recommended and the historical images into a pre-trained image encoder for encoding processing, and feature-encoding the image features obtained by the encoding processing through a projection layer neural network. Mapping to obtain the corresponding image code to be recommended and the historical image code; input the image code to be recommended and the historical image code into the order-independent extractor for generation to obtain a soft prompt tag; pre-construct a hard prompt tag for the information to be recommended, merge the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; map the hidden state feature through a fully connected layer neural network to obtain a topic tag probability distribution of the information to be recommended, and use the topic tag with the largest probability in the topic tag probability distribution as the topic tag recommendation result of the information to be recommended.

[0075] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal topic tag recommendation method based on a large language model provided by the above-mentioned methods, the method comprising: obtaining information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; inputting the images to be recommended and the historical images into a pre-trained image encoder for encoding processing, and performing feature mapping on the image features obtained by the encoding processing through a projection layer neural network to obtain the corresponding image to be recommended code. code and historical image code; input the image code to be recommended and the historical image code into the order-independent extractor for generation to obtain a soft prompt tag; pre-construct a hard prompt tag for the information to be recommended, merge the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; map the hidden state feature through a fully connected layer neural network to obtain a topic tag probability distribution of the information to be recommended, and use the topic tag with the largest probability in the topic tag probability distribution as the topic tag recommendation result of the information to be recommended.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal topic tag recommendation method based on a large language model, characterized in that: The method comprises: Acquire information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; Inputting the image to be recommended and the historical image into a pre-trained image encoder for encoding, and performing feature mapping on the image features obtained by the encoding through a projection layer neural network to obtain corresponding encodings of the image to be recommended and historical images; Inputting the to-be-recommended image code and the historical image code into an order-independent extractor for generation to obtain a soft prompt mark; Pre-constructing a hard prompt tag for the information to be recommended, fusing the soft prompt tag with the hard prompt tag to obtain a mixed tag, and inputting the mixed tag, the text to be recommended, and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; The hidden state features are mapped through a fully connected layer neural network to obtain a topic tag probability distribution of the information to be recommended, and the topic tag with the largest probability in the topic tag probability distribution is used as a topic tag recommendation result of the information to be recommended.

2. The multimodal topic tag recommendation method based on a large language model according to claim 1, characterized in that: The order-independent extractor includes a routing module, a cross-fusion module and a self-fusion module. The step of inputting the image code to be recommended and the historical image code into the order-independent extractor for generation to obtain a soft prompt mark includes: Calling the self-fusion module to perform self-attention mechanism processing on the encoding of the image to be recommended to obtain self-fusion image features; Calling the cross-fusion module to perform cross-attention mechanism processing on the self-fusion image feature and the historical image encoding to obtain a cross-fusion image feature; Calling the routing module to perform routing calculation on the image code to be recommended and the historical image code to obtain a relative attention weight; The cross-fusion image features are weightedly summed by the relative attention weights, and the image features obtained by the weighted sum are fused with the self-fusion image features to obtain soft prompt marks.

3. The multimodal topic tag recommendation method based on a large language model according to claim 2, characterized in that: The self-attention mechanism is performed on the encoding of the image to be recommended by the self-fusion module to obtain the self-fusion image features, including: Obtaining a learnable soft prompt for the information to be recommended; splicing the learnable soft prompt with the image code to be recommended to obtain a spliced ​​image feature; The self-fusion module performs self-attention calculation on the spliced ​​image features to obtain self-fusion image features.

4. The multimodal topic tag recommendation method based on a large language model according to claim 2, characterized in that: The calling of the cross-fusion module to perform cross-attention mechanism processing on the self-fusion image feature and the historical image encoding to obtain the cross-fusion image feature includes: The historical image encoding is used as a key vector and a value vector of cross attention respectively; Using the self-fused image features as a query vector for cross attention; The cross-fusion module performs cross-attention calculation on the key vector, the value vector and the query vector to obtain a cross-fusion image feature.

5. The multimodal topic tag recommendation method based on a large language model according to claim 2, characterized in that: The calling of the routing module to perform routing calculation on the image code to be recommended and the historical image code to obtain a relative attention weight includes: Decomposing the image code to be recommended and the historical image code into a plurality of local representation tags respectively by a decomposer in the routing module; The local representation mark is processed by a cross-attention mechanism through a multi-head cross-attention layer in the routing module to obtain a relative attention weight.

6. The multimodal topic tag recommendation method based on a large language model according to claim 5, characterized in that: The number of the local representation marks is the same as the number of attention heads in the multi-head cross attention layer, and the multi-head cross attention layer in the routing module performs cross attention mechanism processing on the local representation marks to obtain relative attention weights, including: For each first local representation tag of the image to be recommended, mapping the first local representation tag to a cross-attention query feature; Connecting the plurality of second local representation tags encoded by the historical image into a historical tag sequence, and mapping the historical tag sequence into a key feature of cross attention; Performing cross-attention calculation on the query feature and the key feature through the attention head in the multi-head cross-attention layer, and obtaining a head attention weight of the attention head for the first local representation mark; The attention weights calculated for each attention head are used to calculate the mean attention value and obtain the relative attention weight.

7. The multimodal topic tag recommendation method based on a large language model according to claim 1, characterized in that: The topic tag probability distribution of the information to be recommended is predicted by calling a topic tag recommendation model, wherein the topic tag recommendation model includes a pre-trained image encoder, a projection layer network, an order-independent extraction model, a pre-trained large language model, and a fully connected layer network. The training process of the topic tag recommendation model includes: Obtaining a sample of information to be recommended and a sample of historical recommended information, wherein the sample of information to be recommended carries a real recommendation tag corresponding to the topic tag; Inputting the image of the information sample to be recommended and the image of the historical recommended information sample into the pre-trained image encoder and the projection layer network to obtain information sample image features and historical sample image features; Inputting the information sample image features and the historical sample image features into the order-independent extraction model for generation to obtain a predicted soft label; The predicted soft label is fused with the pre-constructed real hard label to obtain a predicted mixed label, and the predicted mixed label, the text of the information sample to be recommended, and the text of the historical recommended information sample are input into the pre-trained large language model for reasoning to obtain a predicted hidden state feature; Mapping the predicted hidden state features through the fully connected layer network to obtain predicted recommendation labels, and constructing a cross entropy loss function based on the predicted recommendation labels and the true recommendation labels; Back propagation is performed in the topic tag recommendation model through the cross entropy loss function to update the parameters of the projection layer network, the order-independent extraction model, and the fully connected layer network.

8. A multimodal topic tag recommendation device based on a large language model, characterized in that: The device comprises: An acquisition module, used to acquire information to be recommended and historical recommendation information, wherein the information to be recommended includes images to be recommended and texts to be recommended for topic tag recommendation, and the historical recommendation information includes historical images and historical texts; An encoding module, used for inputting the image to be recommended and the historical image into a pre-trained image encoder for encoding, and performing feature mapping on the image features obtained by the encoding through a projection layer neural network to obtain corresponding encodings of the image to be recommended and historical image; A generating module, used for inputting the image code to be recommended and the historical image code into an order-independent extractor for generating soft prompt marks; An inference module is used to pre-construct a hard prompt tag for the information to be recommended, fuse the soft prompt tag with the hard prompt tag to obtain a mixed tag, and input the mixed tag, the text to be recommended and the historical text into a pre-trained large language model for inference to obtain a hidden state feature; The recommendation module is used to map the hidden state features through a fully connected layer neural network to obtain a probability distribution of topic tags of the information to be recommended, and use the topic tag with the highest probability in the probability distribution of topic tags as the topic tag recommendation result of the information to be recommended.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal topic tag recommendation method based on a large language model is implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal topic tag recommendation method based on a large language model as described in any one of claims 1 to 7 is implemented.

Images