Sequence recommendation method based on multi-modal feature fusion

Abstract

The invention provides a sequence recommendation method based on multi-modal feature fusion, and relates to the technical field of personalized recommendation. The method comprises the following steps: firstly, extracting multi-modal features according to different modal interaction records of a user; different feature extraction methods are adopted for different modes; then carrying out multi-modal feature fusion and learning by adopting a multi-modal fusion module; constructing a multi-modal fusion sequence recommendation model to perform sequence recommendation; according to the method, by means of an attention mask mechanism, each mode can pay attention to information in the own mode, information interaction between the modes can also be achieved, and rich original information in text and image features is effectively reserved. According to the method, the problem that a traditional multi-modal recommendation model is disordered during feature fusion is solved. Meanwhile, the situation that in the prior art, it is difficult to balance ID feature learning and multi-modal feature learning is improved, and the recommendation ability of the model in the face of long-tail projects and cold start scenes is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of personalized recommendation technology, and in particular to a sequence recommendation method based on multimodal feature fusion. Background Art

[0002] Traditional recommendation system applications include predicting user ratings of items and recommending matching items to users. The rating prediction task is to predict the rating that a user will give to an item, while the item recommendation task is to recommend a set of unobserved / unrated or new items to the target user. Root mean square error (RMSE) and mean absolute error (MAE) are widely used to measure the accuracy of rating predictions, while precision and recall are widely used to evaluate Top-N item recommendations. For explicit ratings, both tasks are applicable, while for implicit ratings, Top-N recommendations are more applicable. Recommendation systems can be roughly divided into three categories: content-based recommendation methods, collaborative filtering (CF) methods, and hybrid methods.

[0003] Models such as convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM) have strong capabilities in processing different types of data such as text and images, and can effectively improve the accuracy and efficiency of recommendations. At the same time, combining different deep learning models can also improve recommendation performance, such as combining CNN and RNN to form a hybrid neural network model to achieve more efficient recommendations.

[0004] It is a common idea to use RNN models to train sequence recommendation models. RNN models can learn transition information similar to Markovchain, and can also learn long-term and short-term sequence information through other methods such as LSTM, or use gated RNN to increase learning ability. Starting from self-attention in the field of NLP, the Transformer model has been applied to the field of recommendation systems in recent years to solve the problem of sequence recommendation. Bert4Rec is based on the Bert model in NLP and considers bidirectional modeling sequences. SSE-PT considers the importance of user embedding in sequence modeling, and proposes the need to use regularization technology stochastic shared embeddings to improve performance. Transformers4Rec also verifies the reliability of Transformer in the recommendation field.

[0005] Existing sequence recommendation methods have deficiencies in feature representation learning. Although Transformer performs well in semantic conversion and representation learning of unimodal features, its advantages are weakened when processing multi-source data (ID, text, image) input. This is because ID features are relatively easy to optimize and have lower training losses in recommendation prediction tasks, while multimodal features provide prior information on project similarity, but when the two are combined, the direction of improvement in evaluation indicators may not be consistent with the direction of loss reduction. Simple fusion strategies such as early fusion or traditional late fusion will cause the lower training loss generated by optimizing ID features to dominate model training during training, making it difficult for the model to fully learn the effective information in multimodal features, limiting the role of multimodal features in recommendation models.

[0006] In practical applications, existing models are unstable when facing different items. In particular, when dealing with new items that do not appear in the training set (cold start problem), ID-based models such as SASRec are difficult to predict accurately. Even if some multimodal models have been improved, the overall performance is still affected by the poor fusion of multimodal and ID features, and cannot effectively meet the diverse recommendation needs. This shows that the existing technology needs to be improved in terms of multimodal information fusion and recommendation accuracy and stability in different project distribution scenarios. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a sequence recommendation method based on multimodal feature fusion in view of the above-mentioned deficiencies in the prior art, so as to enhance the effectiveness of multimodal features in recommendation tasks and improve the accuracy and stability of recommendations.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a sequence recommendation method based on multimodal feature fusion, comprising the following steps: Step 1: Extract multimodal features based on the user's different modal interaction records; use different feature extraction methods for different modalities; Step 1.1: Extract image features; record the image sequence of the user's image interaction with the project and use the pre-trained ViT model to extract image features; Setting Project The image is , image features are obtained through the ViT model : ; Image features It is a matrix composed of multiple eigenvectors, namely: ,in, is the image feature dimension, is the number of blocks the image is split into, represents the global feature representation extracted from the image modality; Step 1.2: Extract text features: For the text sequence of the user's text interaction with the project, use the pre-trained BERT model to extract text features; Setting Project The text is , get text features through the pre-trained BERT model : ; Text Features is a vector with a specific structure, namely: ,in, is the text feature dimension, is the number of text tokens in the text, Represents the global semantic information of the text modality; Step 1.3: Extract ID features; for the ID Seq of the user's ID interaction record for the project, obtain the ID features from the embedding table; Setting Project The ID is , then get the ID feature from the embedding table : , ,in, To embed the table, the ID can be mapped to a specific vector space. is the ID feature dimension; The embedded table By expressing the global features of the image modality extracted in step 1.1 and the text modality global semantic information extracted in step 1.2 Calculate the average generation to unify the dimension of the ID modality with the dimensions of other modalities; Step 1.4: Unify the features of each modality into the same dimension ; The following feature transformation is performed on each modal feature to unify the modal features into the same dimension: ,in, is the feature after the modal feature is transformed in dimension, For modal The linear transformation weight matrix of For modal The bias vector of , Representing modality The transformed target embedding dimension, Representing modality The original embedding dimension of Step 2: Using a multimodal fusion module to perform multimodal feature fusion and learning; the multimodal fusion module consists of a Transformer layer and a DNN layer; Step 2.1: Construct the input features of the multimodal fusion module; Connect the modal features after unifying the dimensions in step 1.4 to obtain the connected features As the input features of the multimodal fusion module: ; Step 2.2: Construct the attention mask matrix of the multimodal fusion module; The original attention mask matrix in the Transformer model is adjusted as follows: According to the number of image block features, ID features, and text tag features, the row and column sizes of the original attention mask matrix are set as ; From the first row to the first row of the attention mask matrix The first Column, assign all elements in it to 1 to ensure the connection feature The image feature part can focus on the ID feature during calculation; From the attention mask matrix From the last row Column, assign all elements in it to 1 to ensure the connection feature The text feature part of can pay attention to the ID feature when calculating; Step 2.3: Input the input features and the adjusted attention mask matrix into the Transformer layer, perform multi-head attention calculation, and obtain the item representation encoded by the Transformer layer; The Transformer layer has Each head performs attention calculation independently. The process of obtaining attention output by one head is as follows: First, a linear transformation layer is introduced to project the input features into a new space; for the input features , whose dimensions are , through the linear transformation matrix , project it into a new space and get the feature ; Next, define the query ,key Sum ; Query ,key Sum By characteristics Through different weight matrices , and Projection is performed, that is: , , ; The attention score is calculated based on the dot product method; the query With key The dot product of the transpose of ;Will Divide by Get the attention score ; Then, the attention score is normalized by the Softmax function to obtain the normalized attention score ; The normalized attention score reflects the sequence Attention weights at different positions; Finally, the attention weights are combined with the values Multiply them together to get i Attention output : ; Concatenate the attention outputs of all heads to get the final attention output of the Transformer layer: ; Output your attention Further feature transformation is performed through a feedforward neural network to obtain the item representation encoded by the Transformer layer; The feedforward neural network consists of two linear layers. The first linear layer is connected by a weight matrix and the bias term The input is transformed, and then nonlinearly transformed through the activation function ReLU to obtain ; Next, the second linear layer passes through the weight matrix and the bias term , and the final output is obtained as the item representation after Transformer layer encoding : ; Step 2.4: The item representation encoded by the Transformer layer is mapped through DNN to obtain the final embedding representation of different modalities; Represent the item after encoding the Transformer layer Recorded as , as shown in the following formula: ,in, is a graphical modal representation of the project, The ID of the project indicates that A text modal representation of an item; Use CLS embedding to represent the global features of images and text; Using two independent DNN layers with LeakyRelu activation layers for different modal representations of items, we get the final embedding representations of different modalities as follows: ,in, For modal The final embedding representation of is the final embedding representation of the modality ID, For modal and the linear transformation weight matrix of ID, is the number of DNN layers, is the bias item corresponding to the DNN layer; Step 3: Build a multimodal fusion sequence recommendation model for sequence recommendation; Step 3.1: Construct multimodal fusion sequence recommendation model input; Set user sequence with item embedding representation for: ,in, Representing modality No. The embedding representation of items, Indicates the modal type; User sequences with item embedding representations As the input of the multimodal fusion sequence recommendation model; Step 3.2: Learning user sequence features; embedding user sequences with item embeddings Input into a series of multi-head self-attention blocks to obtain user sequence features ; The multi-head self-attention block consists of a multi-head self-attention layer using scaled dot products and a two-layer feedforward network ; First, the user sequence with item embedding representation Input to a multi-head self-attention layer using scaled dot products, the output of the multi-head self-attention layer is ; Input query ,key Sum All from ,in, The weight matrices of are independent; Then, the output of the multi-head self-attention layer , input to a two-layer feed-forward network: ,in, is the output of the two-layer feedforward network, are the weight matrices of the two feedforward networks, , is the bias vector of the two-layer feedforward network; It means to connect vectors by row; At the same time, residual connections and layer normalization are used when processing the output of the multi-head attention and the output of the feedforward network; No. A multi-head self-attention block is defined as follows: , ; The first block is defined as and ; Step 3.3: Generate recommendation scores based on user sequence features, cross attention layer, output layer, and activation function; User sequence features The embedded representation of the target project is input into the cross attention layer, and the user features and the target project features are fused through the cross attention layer to mine the correlation between the target project and the user's historical interests; the cross attention layer is an attention network structure; the embedded representation of the target project is set as ; then the key of the cross attention layer input Sum From user sequence features , query From the target item embedding representation ; The weight matrices of are independent; the output of the cross attention layer is ,Will Input into the output layer and the activation function sigmod to obtain the recommendation score of the target item.

[0009] The beneficial effect of adopting the above technical solution is that: the present invention provides a sequence recommendation method based on multimodal feature fusion, which uses the attention mask mechanism to allow each modality to pay attention to the information within its own modality and realize information interaction between modalities, effectively retaining the rich original information in text and image features.

[0010] During the operation of the self-attention mechanism, the ID feature can accurately learn the information contained in the modal feature. At the same time, by limiting the one-way attention of the modal feature to the ID feature, the occurrence of feature assimilation is avoided, ensuring that each feature can give full play to its own advantages during the interaction process, thereby improving the model's understanding and learning ability of the project.

[0011] In terms of user sequence modeling, this method uses an optimized multi-head attention mechanism, which can conduct in-depth analysis of the user's historical behavior sequence, allowing the model to deeply capture the dynamic changes of user preferences over time.

[0012] In addition, this method solves the problem of chaos and disorder in feature fusion in traditional multimodal recommendation models. At the same time, it improves the difficulty in balancing ID features and multimodal feature learning in existing technologies, and enhances the model's recommendation capabilities in the face of long-tail projects and cold start scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a sequence recommendation method based on multimodal feature fusion provided by an embodiment of the present invention; Figure 2 A network architecture diagram of a model for multimodal fusion sequence recommendation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0015] This embodiment takes the Amazon product dataset as an example and adopts the sequence recommendation method based on multimodal feature fusion of the present invention to recommend products to users.

[0016] In this embodiment, a sequence recommendation method based on multimodal feature fusion is provided. Figure 1 , 2 As shown, the following steps are included: Step 1: Extract multimodal features based on the user's different modal interaction records; use different feature extraction methods for different modalities; Step 1.1: Extract image features; record the image sequence of the user's image interaction with the project and use the pre-trained visual Transformer (ViT) model to extract image features; Setting Project The image is , image features are obtained through the ViT model : ; Image features It is a matrix composed of multiple eigenvectors, namely: ,in, is the image feature dimension, is the number of image blocks into which the image is segmented. CLS embedding is a special marker in the Transformer model that is used to represent the global information of the entire input sequence. represents the global feature representation extracted from the image modality.

[0017] Step 1.2: Extract text features: For the text sequence of the user's text interaction with the project, use the pre-trained BERT model to extract text features; Setting Project The text is , get text features through the pre-trained BERT model : ; Text Features is a vector with a specific structure, namely: ,in, is the text feature dimension, is the number of text tokens in the text, Represents the global semantic information of the text modality.

[0018] Step 1.3: Extract ID features; for the ID Seq of the user's ID interaction record for the project, obtain the ID features from the embedding table; Setting Project The ID is , then get the ID feature from the embedding table : , ,in, To embed the table, the ID can be mapped to a specific vector space. is the ID feature dimension; Embedding table of ID features is usually randomly generated, but this is very different from the text features and image features extracted by large pre-trained models such as BERT and ViT (i.e., in the fusion module, the similarity between the query and the key is calculated using the self-attention mechanism). To solve this problem, we represent the image modality global feature extracted in step 1.1 and the text modality global semantic information extracted in step 1.2 Calculate the average to generate the embedding table of ID features , thereby unifying the dimensions of the ID modality with those of other modalities.

[0019] Step 1.4: Unify the features of each modality into the same dimension ; The following feature transformation is performed on each modal feature to unify the modal features into the same dimension: ,in, is the feature after the modal feature is transformed in dimension, For modal The linear transformation weight matrix of For modal The bias vector of , Representing modality The transformed target embedding dimension (uniform dimension), Represents the original embedding dimension of the modality.

[0020] Step 2: Use a multimodal fusion module to perform multimodal feature fusion and learning; the multimodal fusion module is composed of a Transformer layer and a DNN layer; in this embodiment, the multimodal fusion module is responsible for processing and fusing image, text and ID features, and uses an attention mask mechanism to optimize the interaction mode of different modal features of the product to prevent the ID feature from dominating the training loss direction.

[0021] Different from the traditional multimodal Transformer designed for image features and text features, this paper integrates the unique ID feature in the recommendation system into the Transformer. The goal is to obtain a unified framework to transform the embedding of each modality of the project from the original general feature space to a space suitable for recommendation (especially for modality features) and use it in the subsequent recommendation process.

[0022] Step 2.1: Construct the input features of the multimodal fusion module; Connect the modal features after unifying the dimensions in step 1.4 to obtain the connected features As the input features of the multimodal fusion module: .

[0023] Step 2.2: Construct the attention mask matrix of the multimodal fusion module; To prevent the misleading effect of ID embedding on modality embedding, the original attention mask matrix in the Transformer model is adjusted as follows: According to the number of image block features, ID features, and text tag features, the row and column sizes of the original attention mask matrix are set as (Depend on image block features, 1 ID feature, text tag features); From the first row to the first row of the attention mask matrix The first Column, assign all elements in it to 1 to ensure the connection feature The image feature part can focus on the ID feature during calculation; From the attention mask matrix From the last row Column, assign all elements in it to 1 to ensure the connection feature The text feature part of can pay attention to the ID feature during calculation.

[0024] Step 2.3: Input the input features and the adjusted attention mask matrix into the Transformer layer, perform multi-head attention calculation, and obtain the item representation encoded by the Transformer layer; The Transformer layer has Each head performs attention calculation independently. The process of obtaining attention output by one head is as follows: First, a linear transformation layer is introduced to project the input features into a new space; for the input features , whose dimensions are , through the linear transformation matrix , project it into a new space and get the feature ; Next, define the query ,key Sum ; Query ,key Sum By characteristics Through different weight matrices , and Projection is performed, that is: , , .

[0025] The attention score is calculated based on the dot product method; the query With key The dot product of the transpose of ; To prevent the attention score from being affected by the dimension The influence of size leads to numerical instability. Divide by Get the attention score , controlling the attention score to a smaller range.

[0026] Then, the attention score is normalized by the Softmax function to obtain the normalized attention score ; The normalized attention score reflects the sequence Attention weights at different positions; Finally, the attention weights are combined with the values Multiply them together to get Attention output : .

[0027] Concatenate the attention outputs of all heads to get the final attention output of the Transformer layer: .

[0028] Output your attention Further feature transformation is performed through a feedforward neural network to obtain the item representation encoded by the Transformer layer.

[0029] The feedforward neural network consists of two linear layers. The first linear layer is connected by a weight matrix and the bias term The input is transformed, and then nonlinearly transformed through the activation function ReLU to obtain ; Next, the second linear layer passes through the weight matrix and the bias term , and the final output is obtained as the item representation after Transformer layer encoding : .

[0030] Through steps 2.1 to 2.3, the Transformer layer can effectively process the input features and extract rich feature information, providing strong support for subsequent recommendation tasks.

[0031] Step 2.4: The item representation encoded by the Transformer layer is mapped through DNN to obtain the final embedding representation of different modalities; Represent the item after encoding the Transformer layer Recorded as , as shown in the following formula: ,in, is a graphical modal representation of the project, The ID of the project indicates that A text modal representation of an item; CLS embedding is used to represent the global features of images and text; since the features extracted from the pre-trained model already contain position information, there is no need to include additional position embedding in the input embedding.

[0032] In order to obtain better feature representation in the recommendation field, two independent DNN layers with LeakyRelu activation layers are used for the different modal representations of the items, and the final embedding representations of different modalities are obtained as follows: ,in, For modal The final embedding representation of is the final embedding representation of the modality ID, For modal and the linear transformation weight matrix of ID, is the number of DNN layers, is the bias item corresponding to the DNN layer.

[0033] Step 3: Construct a multimodal fusion sequence recommendation model for sequence recommendation; in the sequence recommendation task, user sequence features are generated by user interaction items, and the widely used SASRec sequence recommendation model adopts the attention mechanism to model user sequences. Therefore, the present invention uses SASRec as the backbone network of the sequence recommendation model to learn user sequence features from three input types: image, text, and ID; Step 3.1: Construct multimodal fusion sequence recommendation model input; Set user sequence with item embedding representation for: ,in, Representing modality No. The embedding representation of items, Indicates the modal type; User sequences with item embedding representations As input to the multimodal fusion sequence recommendation model.

[0034] Step 3.2: Learning user sequence features; embedding user sequences with item embeddings Input into a series of multi-head self-attention blocks to obtain user sequence features ; The multi-head self-attention block consists of a multi-head self-attention layer using scaled dot products and a two-layer feedforward network ; First, the user sequence with item embedding representation Input to a multi-head self-attention layer using scaled dot products, the output of the multi-head self-attention layer is ; Input query ,key Sum All from ,in, The weight matrices of are independent; Then, the output of the multi-head self-attention layer , input to a two-layer feed-forward network: ,in, is the output of the two-layer feedforward network, are the weight matrices of the two feedforward networks, , is the bias vector of the two-layer feedforward network; Represents row-wise connection of vectors; the feed-forward network introduces the LeakyReLU function to improve the performance of ReLU in the negative range, making the dynamic range of the model wider and helping to better capture high-order interactions between features.

[0035] At the same time, in order to alleviate the problems of overfitting and instability, residual connections and layer normalization are used when processing the output of multi-head attention and the output of the feedforward network; No. A multi-head self-attention block is defined as follows: , ; The first block is defined as and .

[0036] Step 3.3: Generate recommendation scores based on user sequence features, cross attention layer, output layer and activation function; different from the SASRec model, which directly performs dot product between user sequence features and target item embedding to obtain scores. The embedded representation of the target project is input into the cross attention layer, and the user features and the target project features are fused through the cross attention layer to better explore the correlation between the target project and the user's historical interests; the cross attention layer is an attention network structure; the embedded representation of the target project is set as ; then the key of the cross attention layer input Sum From user sequence features , query From the target item embedding representation ; The weight matrices of are independent; the output of the cross attention layer is ,Will Input into the output layer and the activation function sigmod to obtain the recommendation score of the target item.

[0037] In this embodiment, the multimodal fusion sequence recommendation model is based on the backbone network of the SASRec model, and uses a multi-head attention mechanism to extract the long-term and short-term preferences in the user's historical behavior. The cross-attention module is used to further explore the correlation between user characteristics and target product characteristics to improve the recommendation accuracy.

[0038] In this embodiment, first, the Amazon product data set is preprocessed. The multimodal interaction records between users and products are extracted, including the images, texts and ID information of the products. The data is deduplicated, filled and normalized before multimodal feature extraction. The image block features and global features of the product images are extracted using the pre-trained visual Transformer (ViT) model. The semantic information of the product description text is extracted through the BERT model. The ID vector representation of the product is generated through the embedding table. Then, the features of each modality are unified to the same dimension through linear transformation. The Transformer layer is used to deeply fuse the features of each modality, and the attention mask mechanism is introduced to ensure the complementarity of the modal features. Next, the user's product sequence is obtained based on the historical interaction records of Amazon users, and the user's dynamic interest features are extracted using the improved SASRec model. Finally, the user features and the target product features are fused through the cross-attention mechanism to generate a recommendation score.

[0039] This embodiment also uses the key indicators Recall@10 and NDCG@10 commonly used in the evaluation of recommendation systems to evaluate the method of the present invention. Compared with the existing sequence recommendation model SASRec and the multimodal recommendation models UniSRec and FDSA, the method of the present invention improves by 59.97% and 55.07% compared with SASRec, 14.77% and 15.03% compared with UniSRec, and 54.22% and 49.47% compared with FDSA. In addition, for the indicators Recall@20 and NDCG@20 under the Top20 recommendation task, compared with SASRec, they are improved by 55.62% and 51.65%. Compared with UniSRec, they are improved by 12.69% and 13.08%.

[0040] The data shows that the present invention uses the attention mask mechanism to enable each modality to focus on the information interaction within its own modality and between each other, maintaining the original rich information of text and image features, so that when the self-attention mechanism is running, the ID feature can accurately learn the information in the modal feature, while limiting the one-way attention of the modal feature to the ID feature, avoiding feature assimilation, ensuring that each feature fully exerts its advantages in the interaction, and improving the model's ability to understand and learn projects. In the user sequence modeling link, an optimized multi-head attention mechanism is adopted to deeply analyze the user's historical behavior sequence, so that the model can deeply capture the user's preference dynamics that change over time. It improves the problems of chaos and disorder in feature fusion in traditional multimodal recommendation models, and improves the problems of the existing technology that it is difficult to balance the learning of ID features and multimodal features, and the lack of recommendation when facing long-tail projects and cold start scenarios.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A sequence recommendation method based on multimodal feature fusion, characterized in that: The following steps are involved: Step 1: Extract multimodal features based on the user's different modal interaction records; use different feature extraction methods for different modalities; Step 2: Using a multimodal fusion module to perform multimodal feature fusion and learning; the multimodal fusion module consists of a Transformer layer and a DNN layer; Step 3: Build a multimodal fusion sequence recommendation model for sequence recommendation; Step 3.1: Construct multimodal fusion sequence recommendation model input; Step 3.2: Learning of user sequence features; Input the user sequence with item embedding representation into a series of multi-head self-attention blocks to obtain user sequence features; Step 3.3: Generate recommendation scores based on user sequence features, cross-attention layer, output layer, and activation function.

2. The sequence recommendation method based on multimodal feature fusion according to claim 1, characterized in that: The step 1 comprises: Step 1.1: Extract image features; record the image sequence of the user's image interaction with the project and use the pre-trained ViT model to extract image features ; Step 1.2: Extract text features; Use the pre-trained BERT model to extract text features from the user's text interaction record Text Seq for the project ; Step 1.3: Extract ID features; for the user's ID interaction record for the project, obtain the ID features from the embedding table ; Step 1.4: Unify the features of each modality into the same dimension .

3. The sequence recommendation method based on multimodal feature fusion according to claim 2, characterized in that: The image features It is a matrix composed of multiple eigenvectors, namely: ,in, is the image feature dimension, is the number of blocks the image is split into, represents the global feature representation extracted from the image modality; The text features is a vector with a specific structure, namely: ,in, is the text feature dimension, is the number of text tokens in the text, Represents the global semantic information of the text modality; The ID feature for: , ,in, To embed the table, the ID is mapped to a specific vector space. is the ID feature dimension, For Project ID; The embedded table By representing the global features of the image modality and text modality global semantic information The average generation is calculated to unify the dimensions of the ID modality with those of the other modalities.

4. The sequence recommendation method based on multimodal feature fusion according to claim 3, characterized in that: The step 1.4 performs the following feature transformation on each modal feature to unify each modal feature into the same dimension: ,in, is the feature after the modal feature is transformed in dimension, For modal The linear transformation weight matrix of For modal The bias vector of , Representing modality The transformed target embedding dimension, Representing modality The original embedding dimension of .

5. The sequence recommendation method based on multimodal feature fusion according to claim 4, characterized in that: The step 2 comprises: Step 2.1: Construct the input features of the multimodal fusion module; Connect the features of each modality after unifying the dimensions to obtain the connected features As input features of the multimodal fusion module ; Step 2.2: Construct the attention mask matrix of the multimodal fusion module; The original attention mask matrix in the Transformer model is adjusted as follows: According to the number of image block features, ID features, and text tag features, the row and column sizes of the original attention mask matrix are set as ; From the first row to the first row of the attention mask matrix The first Column, assign all elements in it to 1 to ensure the connection feature The image feature part can focus on the ID feature during calculation; From the attention mask matrix From the last row Column, assign all elements in it to 1 to ensure the connection feature The text feature part of can pay attention to the ID feature when calculating; Step 2.3: Input the input features and the adjusted attention mask matrix into the Transformer layer, perform multi-head attention calculation, and obtain the item representation encoded by the Transformer layer; Step 2.4: The item representation encoded by the Transformer layer is mapped through the DNN to obtain the final embedding representation of different modalities.

6. The sequence recommendation method based on multimodal feature fusion according to claim 5, characterized in that: The step 2.3 comprises: The Transformer layer has Each head performs attention calculation independently. The process of obtaining attention output by one head is as follows: First, a linear transformation layer is introduced to project the input features into a new space; for the input features , whose dimensions are , through the linear transformation matrix , project it into a new space and get the feature ; Next, define the query ,key Sum ; Query ,key Sum By characteristics Through different weight matrices , and Projection is performed, that is: , , ; The attention score is calculated based on the dot product method; the query With key The dot product of the transpose of ;Will Divide by Get the attention score ; Then, the attention score is normalized by the Softmax function to obtain the normalized attention score ; The normalized attention score reflects the sequence Attention weights at different positions; Finally, the attention weights are combined with the values Multiply them together to get i Attention output : ; Concatenate the attention outputs of all heads to get the final attention output of the Transformer layer: ; Output your attention Further feature transformation is performed through a feedforward neural network to obtain the item representation encoded by the Transformer layer; The feedforward neural network consists of two linear layers. The first linear layer is connected by a weight matrix and the bias term The input is transformed, and then nonlinearly transformed through the activation function ReLU to obtain ; Next, the second linear layer passes through the weight matrix and the bias term , and the final output is obtained as the item representation after Transformer layer encoding : .

7. The sequence recommendation method based on multimodal feature fusion according to claim 6, characterized in that: The step 2.4 comprises: Represent the item after encoding the Transformer layer Recorded as , as shown in the following formula: ,in, is a graphical representation of the project, The ID of the project indicates that A text modal representation of an item; Use CLS embedding to represent the global features of images and text; Using two independent DNN layers with LeakyRelu activation layers for different modal representations of items, we get the final embedding representations of different modalities as follows: ,in, For modal The final embedding representation of is the final embedding representation of the modality ID, For modal and the linear transformation weight matrix of ID, is the number of DNN layers, is the bias item corresponding to the DNN layer.

8. The sequence recommendation method based on multimodal feature fusion according to claim 7, characterized in that: The step 3.1 comprises: Set user sequence with item embedding representation for: ,in, Representing modality No. The embedding representation of items, where Indicates the modal type; User sequences with item embedding representations As input to the multimodal fusion sequence recommendation model.

9. The sequence recommendation method based on multimodal feature fusion according to claim 8, characterized in that: The multi-head self-attention block in step 3.2 includes a multi-head self-attention layer using scaled dot products and a two-layer feedforward network ; First, the user sequence with item embedding representation Input to a multi-head self-attention layer using scaled dot products, the output of the multi-head self-attention layer is ; Input query ,key Sum All from ,in, The weight matrices of are independent; Then, the output of the multi-head self-attention layer , input to a two-layer feed-forward network: ,in, is the output of the two-layer feedforward network, are the weight matrices of the two feedforward networks, , is the bias vector of the two-layer feedforward network; It means to connect vectors by row; At the same time, residual connections and layer normalization are used when processing the output of the multi-head attention and the output of the feedforward network; No. A multi-head self-attention block is defined as follows: , , where the first block is defined as and .

10. The sequence recommendation method based on multimodal feature fusion according to claim 9, characterized in that: The step 3.3 comprises: User sequence features The embedded representation of the target project is input into the cross attention layer, and the user features and the target project features are fused through the cross attention layer to mine the correlation between the target project and the user's historical interests; the cross attention layer is an attention network structure; the embedded representation of the target project is set as ; then the key of the cross attention layer input Sum From user sequence features , query From the target item embedding representation ; The weight matrices of are independent; the output of the cross attention layer is ,Will Input into the output layer and the activation function sigmod to obtain the recommendation score of the target item.

Images