An adaptive double-tower multi-modal recommendation method based on knowledge distillation

Abstract

The application provides a kind of adaptive double-tower multi-modal recommendation method based on knowledge distillation, it is related to multi-modal recommendation technical field, the application is by collecting and obtaining user-item interaction information and item multi-modal information, data cleaning and format conversion are carried out;Teacher-student model is used to carry out the knowledge distillation of multi-modal information;Based on GNN and expert network, a double-tower model is constructed;The preference score of the interaction tower and the content tower under the double-tower model is calculated respectively;The double-tower preference score is mixed based on scene characteristics;After calculating the preference score, the item recommendation is carried out.The application is significantly better than the traditional single-modal recommendation algorithm.This progress not only reflects the performance improvement of the recommendation system, but also has obvious advantages in space and computing resource saving.Therefore, the application has important application value and broad market prospect in the field of multi-modal recommendation system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal recommendation, and particularly relates to a self-adaptive double-tower multi-modal recommendation method based on knowledge distillation. BACKGROUND

[0002] In recent years, with the continuous development of information technology, a large amount of information is flooding on the Internet, and some scholars have realized the necessity of using multi-modal information in personalized recommendation. Multi-modal information can provide rich semantic information, and through the exploration of modal preference, the recommendation algorithm can provide more rich and diversified recommendation content for users, increase the visual attraction and content attraction of items to users, and the model can more comprehensively understand the user preference; the lack of historical interaction data of new users or new items causes the cold start problem, and since there is a lack of data, the recommendation algorithm cannot accurately predict the user interest, which reduces the recommendation quality and seriously affects the user satisfaction. Especially in the e-commerce platform, inaccurate recommendation reduces the click rate and purchase rate, which has a great negative impact on the platform revenue. The addition of multi-modal information optimizes the algorithm for user preference exploration, so the addition of multi-modal information has great necessity in the field of personalized recommendation.

[0003] The current multi-modal recommendation technology difficulty mainly lies in multi-modal data preprocessing and multi-modal data and behavior data fusion mode. The current fusion mode mainly includes the following:

[0004] Multi-modal data preprocessing: mainly using CLIP and BEiT3 large models. CLIP is a kind of multi-modal pre-training large model, which embeds images and texts into a shared multi-modal space through joint training of a large number of image and text pairs, and realizes the same content of image and text with similar representation through contrastive learning of a large number of image and text pairs. CLIP can be used for image search, image generation and personalized recommendation, etc., realizing the semantic matching of natural language description and image content, and greatly improving the performance of multi-modal recommendation system. BEiT-3 realizes the most advanced migration performance on multiple visual and visual-linguistic tasks, including object detection, instance segmentation, semantic segmentation, image classification, visual reasoning, visual question answering, picture description generation and cross-modal retrieval, etc. By regarding image as a kind of "foreign language", the model is quickly and uniformly processed through mask "language" modeling of image, text and image-text pairs. Especially, a unified mask data modeling (Masked Data Modeling) is adopted as the pre-training target, and this method allows the model to learn the features of images and texts by predicting the masked parts.

[0005] The fusion mode of multi-modal data and behavior data can be mainly divided into the following categories:

[0006] 1. Simple Concatenation The most common method is to simply concatenate the feature vectors of modal data and behavior data. First, feature extraction is performed: different models are used to extract the features of data in each modality, and then a sequence model is used to extract behavior features. Then the above feature vectors are directly concatenated into a long vector. The advantage of this method is simple and direct, but the disadvantage is that the relationship between features of different modalities may not be fully explored and utilized.

[0007] 2. Fixed Weighting Assign fixed weights to the features of each category, and then obtain a comprehensive feature representation by weighted summation. This method is relatively simple to calculate, easy to implement and understand, and since the weights are pre-set, there is no need to repeatedly calculate during the iteration process, so it usually has high computational efficiency, especially for large-scale data sets.

[0008] 3. Graph Neural Networks (GNN) Graph neural networks can model users, items, and their interaction relationships as a graph structure, and then propagate and aggregate on the graph. The user and item and their multi-modal features are constructed into a graph structure, and then graph convolution operation is performed for feature propagation and aggregation to better learn the representation of nodes. Finally, the node embedding in the graph neural network is used as the embedding of the user and item.

[0009] Current multi-modal recommendation technology has major defects in handling the above problems:

[0010] Defects in multi-modal data preprocessing:

[0011] Although large language models have shown great potential in multi-modal recommendation systems, there are still some significant defects: High demand for computing resources: hardware-wise, the CLIP model requires a large number of high-performance GPUs or TPUs for training. The training process may involve billions of parameters and require processing large-scale image-text pair datasets; storage-wise, large-scale datasets and intermediate model checkpoints require a large amount of storage space. Processing and storing these large amounts of data require high-performance storage devices. Real-time aspect: in online recommendation scenarios, the system needs to respond within a very short time. The complexity and computational requirements of large models may cause the response time to be prolonged, making it impossible to meet the real-time requirements. Availability aspect: in a high-concurrency environment (such as large e-commerce websites or video platforms), the system needs to handle a large number of user requests simultaneously. The high computational requirements of the CLIP model may cause the system to be unable to efficiently handle concurrent requests, thereby affecting user experience.

[0012] Data utilization method:

[0013] In modern recommendation systems, multi-modal information such as images and text descriptions has become an important way to supplement the characteristics of recommendation system items, but how to better utilize the above information to fully integrate it with user behavior information has become an important knowledge that needs to be explored. The existing method blindly integrates the two kinds of information, causing information aliasing and loss of modal information,

[0014] Simple splicing defects: simple splicing of interaction embedding and multi-modal embedding greatly increases the dimension of the feature vector, leading to increased computational complexity and possible "dimension disaster" problems. The spliced feature vector may contain a large amount of redundant information, increasing the burden on the model and making it difficult to extract effective information.

[0015] Fixed weighting defects: fixed weights cannot adapt to the dynamic changes and complementarity between modalities in different data samples, resulting in the model being unable to fully utilize the potential of multi-modal data, ignoring the unique contribution and importance of each modality, and possibly leading to some key information being underestimated, resulting in the model's generalization ability being weak.

[0016] GNN defects: interaction embedding and multi-modal embedding are treated as nodes in GNN for graph convolution operation, and interaction information and multi-modal information propagate in the same GNN network, with the smoothing effect of graph convolution operation gradually increasing, resulting in loss of semantic features of modal information. SUMMARY

[0017] To overcome the shortcomings of the prior art, the present application provides a self-adaptive double-tower multi-modal recommendation method based on knowledge distillation;

[0018] A self-adaptive double-tower multi-modal recommendation method based on knowledge distillation, comprising the following steps:

[0019] Step S1: obtaining user-item interaction information and item multi-modal information;

[0020] Step S2: using a teacher-student model for multi-modal information knowledge distillation;

[0021] Step S2.1: selecting a teacher model;

[0022] Selecting a CLIP model as the teacher model for multi-modal feature extraction of image data and text data, and the teacher model processing function is denoted as f T :

[0023] T image =f T (X image )

[0024] T text =f T (X text )

[0025] wherein the item image data is represented as The item text data is represented as wherein are the original image data and original text data of the Rth item, respectively, R is the total number of items; after the processing of the teacher model, the image data and text data results become and wherein are the image data and text data of the Rth item after the processing of the teacher model, respectively.

[0026] Step S2.2: Selecting the student model; wherein the student image modal processor selects the lightweight convolutional network MobileNet, and the processing function is denoted as f S,i , and the student text modal processor selects the BERT model, and the processing function is denoted as f S,t ;

[0027] S image = f S,i (X image )

[0028] S text = f S,t (X text )

[0029] After the processing of the student model, the image data and text data, i.e., the student image modal embedding S image and the student text modal embedding S text , are represented as and wherein are the image data and text data of the Rth item after the processing of the student model.

[0030] Step S2.3: Knowledge distillation preparation: through knowledge distillation, the student can better imitate the structure of the teacher, and a loss function is designed;

[0031] Contrastive loss: the same item representation processed by the teacher model and the student model is defined as a positive sample, and different item representations are defined as negative samples; wherein the image modal positive sample is defined as The text modal positive sample is defined as

[0032]

[0033]

[0034] L CD = L CDI + LCDT

[0035] where L CDI is the image modality distillation contrast loss, L CDT is the text modality distillation contrast loss, L CD is the total distillation contrast loss, and τ is the temperature parameter.

[0036] Distribution loss: Let be the predicted probability distribution of the teacher model for the original image feature X i , be the predicted probability distribution of the teacher model for the original text feature T i ; be the predicted probability distribution of the student model for the original image feature X i , be the predicted probability distribution of the student model for the original text feature T i :

[0037]

[0038]

[0039] L KL = L KLI + L KLT

[0040] where L KLI is the image modality distillation distribution loss, L KLT is the text modality distillation distribution loss, and L KL is the total distillation distribution loss.

[0041] MLM loss: Introduce auxiliary loss, specifically use mask guidance, randomly select some positions in the input sequence for mask processing, replace the values of these positions with special markers [MASK]; through the student model, predict the original values of these mask positions; compare the predictions of the student model for the mask positions with the actual values, and calculate the prediction error:

[0042] P MLM (m j ) = softmax(f S (X j-mask ) · f T (X j ))

[0043]

[0044] where X j-mask is the mask input sequence, and X jis the original input sequence, M is the set of masked positions, where the j-th masked position is denoted as m j , E is the embedding matrix, P MLM is the prediction distribution of the MLM task, P MLM (m j ) is the probability that the student model predicts the correct label m j at position j;

[0045] The above losses are integrated as follows:

[0046] L=L CE +αL KL +βL MLM

[0047] where α, β are loss coefficients; by minimizing the above loss, the student model is realized to imitate the teacher model;

[0048] Step S3: constructing a double-tower model based on the GNN and the expert network;

[0049] Step S3.1: constructing an interaction tower using the GNN;

[0050] Define a bipartite graph G=(V, E), where the node set is defined as V, V=U∪I, U is the set of user nodes, I is the set of item nodes, the number of nodes is |V|, the edge set is E, and the number of edges is |E|; if the user node and the item node in the bipartite graph have interaction, then there is a corresponding edge between the two nodes, that is, E={(u, i)|u∈U, i∈I, R ui ≠0}; R ui is the rating of user u to item i; use to represent the node embedding of the l-th layer bipartite graph, where d is the dimension of each node embedding, and the propagation formula of the ID embedding is as follows:

[0051]

[0052]

[0053] where σ() is the sigmoid function, A∈R |V|×|V| is the adjacency matrix of the interaction graph, is the adjacency matrix of the interaction graph after adding self-loops, D is the diagonal matrix, is the adjacency matrix of the interaction graph after adding self-loops, E ID is the final interaction tower embedding, where the interaction tower embedding E ID contains two parts: the interaction tower user embedding E ID,u and the interaction tower item embedding E ID,i .

[0054] Step S3.2: Constructing the content tower based on the expert network, and performing modal information fusion;

[0055] The student image modal embedding S image and the student text modal embedding S text are obtained by knowledge distillation of the teacher-student model. CG-o The spliced user tower original input E CG-o is obtained, and the calculation formula is as follows:

[0056] E image = [S text ]

[0057] The teacher-student model is the entire process of distillation learning using the teacher model and the student model respectively.

[0058] In order to better fuse multiple modalities, modal expert networks MMMoE are used to design different experts for image modalities, text modalities, and text-image fusion modalities respectively. Different expert networks model different aspects of data, and then assign different weights to each expert task through a gating fusion mechanism. Finally, the expert results are dynamically fused according to the gating network weight distribution, so as to perform more detailed feature fusion. The project features of the modal expert network MMMoE after fusion are taken as the content tower project embedding:

[0059]

[0060] where n exp is the number of experts, Expert k () is the calculation function of the kth expert network, and Expert k (), Gate(), and ffn() are all two-layer MLPs.

[0061] The content tower user embedding E CG,u is obtained by traversing and aggregating the projects in the content tower by querying the user-project interaction records.

[0062]

[0063] where N u is the neighbor of user u, |N u | is the number of neighbors of user u, N i is the neighbor of project i, |N i | is the number of neighbors of project i, e CG,u and e CG,i are the embedding representations of the user nodes and the project nodes under the content tower respectively. The user tower embedding E CG,u is obtained by splicing the content tower user embedding E CG,i and the content tower project embedding E CG;

[0064] Step S4: Calculate the preference scores of the interaction tower and the content tower under the double tower model respectively;

[0065] A scene-aware double tower preference mixing layer is proposed; the double tower preference mixing layer is used to calculate the mixed user-item preference under the influence of the ID graph and the CG graph;

[0066] The ID graph user-item preference is:

[0067] f ID (u,i)=e ID,u ·e ID,i

[0068] Where e ID,u ,e ID,i are the node representation of the user in the ID graph and the item node representation in the ID graph respectively; f ID (u,i) is the preference score obtained by calculating the similarity between user items in the ID graph.

[0069] The CG graph user-item preference is:

[0070] f CG (u,i)=e CG,u ·e CG,i

[0071] Where e CG,u ,e CG,i are the node representation of the user in the ID graph and the item node representation in the ID graph respectively. f CG (u,i) is the preference score obtained by calculating the similarity between user items in the CG graph.

[0072] Step S5: Mix the double tower preference scores based on the scene features:

[0073] The scene features include device type, time; multi-hot encoding is performed on the scene features and combined to obtain the complete scene feature X sf , the weight w is calculated based on the scene feature, and the calculation method is as follows:

[0074] z=W2(W1X sf +b1)+b2

[0075]

[0076] Where W1 and W2 are the fitted weight matrix, b1 and b2 are the bias, and the two preference scores are combined through the weight w to obtain the final user-item preference score:

[0077] f(u,i)=w·f ID(u,i)+(1-w)·f CG (u,i)

[0078] Step S6: Calculate the preference score to recommend the project.

[0079] By comparing the preference score f(u,i), the top K projects are finally selected as the final recommendation result for recommendation, which can best fit the user preference and recommend the projects of interest to the user.

[0080] The beneficial effects produced by the above technical solutions are as follows:

[0081] The present application provides a kind of adaptive double tower multi-modal recommendation method based on knowledge distillation, which has significant advantages in multi-modal data processing.The present application effectively carries out multi-modal feature preprocessing by introducing knowledge distillation mechanism, reduces the calculation pressure of model.At the same time, the present application uses double tower structure to process different types of data independently, ensures the independence and integrity between each modal feature and behavior feature.More importantly, the adaptive mechanism of the present application can dynamically adjust the weight distribution between modes according to scene characteristics, so that the recommendation result is more personalized and accurate.Compared with the traditional fixed weight fusion method, the present application can more effectively utilize multi-modal data, and provide more rich and accurate recommendation result.

[0082] In experimental test, the present application is significantly better than traditional single modal recommendation algorithm.This progress not only reflects the performance improvement of recommendation system, but also has obvious advantages in space and computing resource saving.Therefore, the present application has important application value and broad market prospect in the field of multi-modal recommendation system. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 It is the whole flow chart of adaptive double tower multi-modal recommendation method in the embodiment of the present application;

[0084] Figure 2 It is the schematic diagram of teacher-student knowledge distillation in the embodiment of the present application;

[0085] Figure 3 It is the schematic diagram of user tower expert network in the embodiment of the present application. DETAILED DESCRIPTION

[0086] The specific embodiments of the present application will be further described in detail below in combination with the drawings and examples.The following examples are used to illustrate the present application, but not to limit the scope of the present application.

[0087] A kind of adaptive double tower multi-modal recommendation method based on knowledge distillation, as shown in Figure 1 It includes the following steps:

[0088] Step S1: obtaining user-item interaction information and item multi-modal information;

[0089] Step S2: using a teacher-student model to perform knowledge distillation of multi-modal information, as shown in Figure 2 ;

[0090] Step S2.1: selecting a teacher model;

[0091] The CLIP model is selected as the teacher model for multi-modal feature extraction of image data and text data, and the teacher model processing function is denoted as f T :

[0092] T image =f T (X image )

[0093] T text =f T (X text )

[0094] Wherein the item image data is represented as The item text data is represented as Wherein The Rth item original image data and original text data are respectively and Wherein The Rth item image data and text data processed by the teacher model are respectively

[0095] Step S2.2: selecting a student model;

[0096] The selected student model should have lower computational complexity in order to quickly infer in actual application. At the same time, the student model should be small in order to reduce memory occupation and deployment cost. Finally, the performance of the student model should be close to that of the teacher model in terms of accuracy, precision, recall, etc. Therefore, the student image modality processor selects the lightweight convolutional network MobileNet, and the processing function is denoted as fS,i , and the student text modality processor selects the BERT model, and the processing function is denoted as f S,t ;

[0097] S image =f S,i (X image )

[0098] S text =f S,t (X text )

[0099] image data and text data of the Rth project after being processed by the student model. image and student text modality embedding S text , respectively. and wherein is the image data and text data of the Rth project after being processed by the student model.

[0100] Step S2.3: knowledge distillation preparation;

[0101] The multi-modal pre-training large model has learned good project multi-modal representations, and then through knowledge distillation, the student can better imitate the structure of the teacher. The following loss function is designed:

[0102] Contrastive loss: the same project representation processed by the teacher model and the student model is defined as a positive sample, and different project representations are defined as negative samples; wherein the image modality positive sample is defined as The text modality positive sample is defined as

[0103]

[0104]

[0105] L CD = L CDI + L CDT

[0106] wherein L CDI is the image modality distillation contrastive loss, L CDT is the text modality distillation contrastive loss, L CD is the total distillation contrastive loss, and τ is a temperature parameter.

[0107] Distribution loss: let be the prediction probability distribution of the teacher model for the original image feature X i , be the prediction probability distribution of the teacher model for the original text feature T i ; be the prediction probability distribution of the student model for the original image feature X i , be the prediction probability distribution of the student model for the original text feature T i :

[0108]

[0109]

[0110] LKL = L KLI + L KLT

[0111] wherein L KLI is an image modality distillation distribution loss, L KLT is a text modality distillation distribution loss, L KL is a total distillation distribution loss;

[0112] MLM loss: auxiliary loss is introduced to help the student model learn better. Mask guide is used to randomly select some positions in the input sequence for masking, and the values of these positions are replaced with special markers [MASK]; the student model predicts the original values of these masked positions; the prediction error of the student model on the masked positions is calculated by comparing the prediction of the student model on the masked positions with the actual values:

[0113] P MLM (m j ) = softmax(f S (X j-mask ) · f T (X j ))

[0114]

[0115] wherein X j-mask is a masked input sequence, X j is an original input sequence, M is a set of masked positions, wherein the jth masked position is marked as m j , E is an embedding matrix, P MLM is the prediction distribution of the MLM task, P MLM (m j ) is the probability of the student model predicting the correct label m j at position j;

[0116] The above losses are integrated:

[0117] L = L CE + aL KL + bL MLM

[0118] wherein a, b are loss coefficients; by minimizing the above loss, the student model imitates the teacher model; thereby maintaining high performance in an environment with limited computing resources.

[0119] Step S3: build a double tower model based on GNN and expert network, as shown in Figure 3 ;

[0120] Step S3.1: build an interaction tower using GNN;

[0121] A bipartite graph G=(V,E) is defined, where the node set is defined as V, V=U∪I, U is a set of user nodes, I is a set of item nodes, the number of nodes is |V|, and the edge set is E, the number of edges is |E|; if the user node and the item node in the bipartite graph have interaction, then there is a corresponding edge between the two nodes, that is, E={(u,i)|u∈U,i∈I,R ui ≠0};R ui is the rating of the user u to the item i; the node embedding of the l-th layer bipartite graph is represented by , where d is the dimension of each node embedding. Inspired by LightGCN, the propagation formula of the ID embedding is as follows:

[0122]

[0123]

[0124] where σ() is a sigmoid function; the above formula realizes the consideration of the features of the node itself and the features of the neighbor nodes, enriching the representation of the node. A∈R |V|×|V| is the adjacency matrix of the interaction graph, is the adjacency matrix of the interaction graph after adding self-loops, D is a diagonal matrix, is the adjacency matrix of the interaction graph after adding self-loops, E ID is the final interaction tower embedding, and the calculation method is to calculate the average value of the node embedding of the first l-layer bipartite graph; wherein the interaction tower embedding E ID contains two parts: the interaction tower user embedding E ID,u and the interaction tower item embedding E ID,i .

[0125] Step S3.2: constructing a content tower based on an expert network and performing modal information fusion;

[0126] The student image modal embedding S image and the student text modal embedding S text are spliced to obtain the user tower original input E CG-o , and the calculation formula is as follows:

[0127] E CG-o =[S image ,S text ]

[0128] The teacher-student model is the entire process of distillation learning using the teacher model and the student model respectively;

[0129] To better fuse multiple modalities, a modal expert network MMMoE is used to design different experts for image modalities, text modalities and text-image fusion modalities respectively; different expert networks model different aspects of data, and then assign different weights to each expert task through a gating fusion mechanism, and finally dynamically fuse the expert results according to the gating network weight distribution, so as to perform more fine-grained feature fusion. The modal expert network MMMoE simultaneously processes multiple modalities to greatly improve the generalization ability of the model. The fused item features of the modal expert network MMMoE are used as content tower item embeddings:

[0130]

[0131] where n exp is the number of experts, Expert k () is the calculation function of the kth expert network, where Expert k (), Gate(), ffn() are all two-layer MLPs.

[0132] The content tower is traversed and aggregated for the items in the content tower by querying the user-item interaction records to obtain content tower user embeddings E CG,u

[0133]

[0134] where N u is the neighbor of user u, |N u | is the number of neighbors of user u, N i is the neighbor of item i, |N i | is the number of neighbors of item i, e CG,u , e CG,i are the embedding representations of the user nodes and item nodes in the content tower respectively. Concatenating the content tower user embeddings E CG,u and the content tower item embeddings E CG,i yields the final user tower embeddings E CG ;

[0135] Step S4: Calculate the preference scores of the interaction tower and the content tower under the dual tower model respectively.

[0136] A scene-aware dual tower preference mixing layer is proposed; the dual tower preference mixing layer is used to calculate the mixed user-item preference under the influence of the ID graph and the CG graph;

[0137] The ID graph user-item preference is:

[0138] f ID (u, i) = e ID,u · e ID,i

[0139] where eID,u , e ID,i are the node representation of users in the ID graph and the item node representation in the ID graph, respectively.f ID (u, i) is the preference score obtained by calculating the similarity between user items in the ID graph.

[0140] The CG graph user-item preference is:

[0141] f CG (u, i) = e CG,u ·e CG,i

[0142] where e CG,u , e CG,i are the node representation of users in the ID graph and the item node representation in the ID graph, respectively.f CG (u, i) is the preference score obtained by calculating the similarity between user items in the CG graph.

[0143] Step S5: Mixing of double-tower preference scores based on scene features:

[0144] The scene features include device type and time; the device type can be a discrete classification feature, such as mobile phone, tablet, computer, etc.; the time is the time of the day, which can be a discrete numerical feature, such as 20:40. The scene features are multi-hot encoded and combined to obtain the complete scene feature X sf The weight w is calculated based on the scene features, and the calculation method is as follows:

[0145] z = W2(W1X sf +b1) + b2

[0146]

[0147] where W1 and W2 are the fitted weight matrices, b1 and b2 are the bias, and the two preference scores are combined through the weight w to obtain the final user-item preference score:

[0148] f(u, i) = w·f ID (u, i) + (1-w)·f CG (u, i)

[0149] Unlike ordinary single-channel multi-modal recommendation, the double-tower model dynamically combines the ID prediction scores and CG prediction scores generated by parallel modeling, which is more effective and accurate.

[0150] Step S6: Calculate the preference score to make item recommendations.

[0151] By comparing the preference scores f(u, i), the first K items are finally selected as the final recommendation result for recommendation, that is, the user preference is fitted to the maximum extent, and the items interested by the user are recommended.

[0152] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology applied. It should be understood by those skilled in the art that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features are replaced with each other to form a technical solution with similar functions disclosed in the embodiments of the present disclosure (but not limited to).

Claims

1. An adaptive dual-tower multimodal recommendation method based on knowledge distillation, characterized in that, Includes the following steps: Step S1: Obtain user-project interaction information and project multimodal information; Step S2: Use a teacher-student model to perform knowledge distillation of multimodal information; Step S3: Construct a dual-tower model based on GNN and expert network; Step S4: Calculate the preference scores for the interaction tower and content tower under the dual-tower model respectively; Step S5: Perform dual-tower preference score fusion based on scene features; Step S6: Calculate preference scores and then recommend items; Step S2 includes the following steps: Step S2.1: Select the teacher model; The CLIP model was chosen as the teacher model for multimodal feature extraction from image and text data. The teacher model processing function is denoted as... : ， ； The project image data is represented as follows: The project text data is represented as ,in , The original image data and original text data for the R-th item are given, respectively, where R is the total number of items. After processing by the teacher model, the image data and text data results are transformed into... and ,in , These are the image data and text data of the Rth item after processing by the teacher model, respectively; Step S2.2: Select the student model; where the student image modality processor is the lightweight convolutional network MobileNet, and the processing function is denoted as... The student text modality processor uses the BERT model, and the processing function is denoted as... ; ， ； The image and text data processed by the student model, i.e., student image modality embedding. Student text modal embedding , respectively represented as and ,in , The image and text data of the Rth project after processing by the student model; Step S2.3: Knowledge Distillation Preparation: Through knowledge distillation, students can better imitate the teacher's structure and design loss functions; The loss mentioned in step S2.3 specifically includes: Contrast loss: Representations of the same item processed by both the teacher and student models are considered positive samples, while representations of different items are considered negative samples; where positive samples in the image modality are defined as... Text modality positive samples are defined as ; ， ， ； in For image modal distillation contrast loss, For text modal distillation contrast loss, The total distillation contrast loss is represented by τ, where τ is the temperature parameter. Distributed loss: Let For the teacher model, the original image features The predicted probability distribution For the teacher model, the original text features The predicted probability distribution; For student models, the original image features The predicted probability distribution For student models, the original text features The predicted probability distribution: ， ， ； in For image modal distillation distribution loss, For text modal distillation distribution loss, This represents the total distillation distribution loss; MLM Loss: An auxiliary loss is introduced, specifically using masking. Randomly selected positions in the input sequence are masked, and the values ​​at these positions are replaced with a special marker [MASK]. The student model then predicts these masked positions to recover their original values. The predictions of the student model are compared with the actual values ​​to calculate the prediction error. ， ； in The input sequence is a mask. Let M be the original input sequence, and M be the set of masked positions, where the position of the j-th mask is labeled as... E is the embedding matrix. It is the predicted distribution of the MLM task. The student model correctly predicts the label m at position j. j The probability of; In summary, the above losses are as follows: ； Where α and β are loss coefficients; by minimizing the above loss, the student model achieves the imitation of the teacher model; Step S3 includes the following steps: Step S3.1: Construct an interaction tower using GNN; Define a bipartite diagram The set of nodes is defined as follows: U is the set of user nodes, I is the set of project nodes, and the number of nodes is... The edge set is The number of sides is If a user node and a project node in a bipartite graph interact, then there is a corresponding edge between the two nodes, i.e. ;R ui Rate user u for item i; use Let represent the node embeddings of the l-th layer bipartite graph, where d is the dimension of each node embedding, and the propagation formula for ID embedding is as follows: ， ； Where σ() is the sigmoid function, This is an adjacency matrix of the interaction graph. To form the adjacency matrix of the interaction graph after adding self-loops, For the angle matrix, Adjacency matrix of the interaction graph after adding self-loops For the final interactive tower embedding, where the interactive tower embedding It consists of two parts: Interactive Tower User Embedding Embedded with Interactive Tower Project ; Step S3.2: Construct a content pyramid based on the expert network and perform modal information fusion; Step S3.2 specifically involves embedding the student image modality obtained by knowledge distillation of the teacher-student model. Student text modal embedding The original input of the user tower is obtained by splicing. The calculation formula is as follows: ； The teacher-student model refers to the entire process of distillation learning using both the teacher model and the student model. To better integrate multiple modalities, a Modal Expert Network (MMMoE) is used to design different experts for image modality, text modality, and text-image fusion modality. Different expert networks model different aspects of the data, and then a gating fusion mechanism is used to assign different weights to each expert task. Finally, the dynamic fusion expert results are allocated based on the gating network weights, resulting in more refined feature fusion. The fused project features from the MMMoE are then embedded as content tower projects. ； in For the number of experts, Let be the computation function of the k-th expert network, where Both are two-layer MLPs; By querying user-project interaction records, we can traverse and aggregate projects in the content tower to obtain the embedded user data of the content tower. : ； in user Neighbors For users The number of neighbors, The neighbor of project i Let be the number of neighbors of project i. , These represent the embedded representations of user nodes and project nodes under the content tower, respectively; concatenating the content tower user embeddings. Embedded with Content Tower Project That is, to obtain the end-user tower embedding ; Step S4 specifically involves proposing a scene-aware dual-tower preference hybrid layer; the dual-tower preference hybrid layer is used to calculate the hybrid user-item preferences jointly influenced by the ID graph and CG graph; IDGraph user-project preferences are: ； in These represent the user node representation and the project node representation in the ID diagram, respectively. To calculate the preference score obtained from the similarity between user items in the ID graph; CG image user - project preference is: ； in These represent the user node representation and the project node representation in the ID diagram, respectively. To calculate preference scores based on the similarity between user items in the CG graph; The scene features mentioned in step S5 include device type and time; the scene features are multi-hot encoded and combined to obtain the complete scene features. The weight w is calculated based on scene features, and the calculation method is as follows: ， ； Where W1 and W2 are the fitted weight matrices, and b1 and b2 are the biases, the two preference scores are combined by weighting φ to obtain the final user-item preference score: ； In step S6, preference scores are compared. Finally, the top K items are selected as the final recommendation results, which can best fit user preferences and recommend items that users are interested in.

Images