A knowledge distillation multimodal recommendation method based on modal decoupling

Abstract

The present invention relates to the field of personalized recommendation technology, and discloses a knowledge distillation multimodal recommendation method based on modal decoupling. For a specific user, a trained multimodal recommendation model can recommend items. The training process of the multimodal recommendation model includes: extracting text features corresponding to the text information of the item, extracting visual features corresponding to the image information of the item; passing the text features through a text teacher network and a text student network respectively; passing the visual features through a visual teacher network and a visual student network; and then implementing the training of the multimodal recommendation model through attention mechanism, contrastive learning, knowledge distillation and other technologies. The teacher network uses rich semantic information for supervised learning, thereby more comprehensively capturing multimodal data. Through knowledge distillation, the information in the teacher network is smoothly transferred to the student network, effectively solving the data sparsity problem in the recommendation system.

Description

Technical Field

[0001] The present invention relates to the technical field of personalized recommendation, and in particular to a knowledge distillation multimodal recommendation method based on modal decoupling. Background Art

[0002] Recommendation systems have become an essential component of various digital platforms such as e-commerce, social media, and streaming services, and play a very important role in news recommendation, short video recommendation, and e-commerce business. Recommendation systems provide users with personalized content by accurately analyzing and predicting user behavior and preferences, which has profoundly changed user engagement and experience. This personalized recommendation not only improves user satisfaction, but also brings higher user retention and activity to the platform. Among the many strategies used by recommendation systems, collaborative filtering (CF) is considered to be a basic and effective technology. This technology predicts content that users may be interested in by leveraging and analyzing the similarities in behavior and preferences between users.

[0003] However, although collaborative filtering technology performs well in many cases, it also has its limitations. Models driven by collaborative filtering often face challenges when faced with sparse user-item interaction data. This is because in this case, the model often finds it difficult to accurately capture the user's true preferences, resulting in recommendations that may be inaccurate or not meet the user's true needs. This problem is particularly prominent in the recommendation of new users or new items, because the user-item interaction data in these cases is usually more sparse. Therefore, how to improve the performance of recommendation systems when faced with sparse data is an important direction of current research.

[0004] To address these limitations, existing technologies have begun to shift their focus to multimodal recommender systems. Multimodal recommender systems make full use of a large amount of auxiliary information, such as user attributes, reviews, and images, to more accurately predict user preferences and item characteristics. However, the shift to multimodal recommendation models brings a series of new challenges. Extracting meaningful features from multiple modalities is crucial, but it is undoubtedly a difficult task due to the high dimensionality and complexity of multimodal data. Traditional shallow neural networks (SNNs) often perform poorly in this area and are unable to gain deep insights into complex modalities. This has driven the application of deep neural networks (DNNs), which have stronger representation learning capabilities. However, in order to improve the effectiveness of deep neural networks, a large amount of training data is required, and the data sparsity in the recommender system environment hinders the training of deep neural networks and may have an adverse impact on the performance of the recommender system. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a knowledge distillation multimodal recommendation method based on modal decoupling, and applies it to the recommendation system domain model; by decoupling the information extracted from different modalities, the similar features and specific features between different modalities are characterized, and the information learned in the teacher network is transferred to the student network through knowledge distillation, so as to better perform vector representation of different models.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A knowledge distillation multimodal recommendation method based on modal decoupling recommends items to specific users through a trained multimodal recommendation model. The training process of the multimodal recommendation model includes:

[0008] Step 1: Extract the text features corresponding to the text information of the item , extract the visual features corresponding to the image information of the item ;No. The semantic label vector of an item is ;

[0009] Step 2: Text features The text teacher network and the text student network are respectively passed through; the text teacher network extracts high-level information of the text through a deep neural network ; The text student network extracts the full information of the text modality through a shallow neural network Similar information to text cross-modality Then perform subtraction to obtain text modality specific information ;Will Input to classifier 1 , get the first vector ,based on and The structural loss of ;

[0010] Step 3: Visual features After the visual teacher network and the visual student network; the visual teacher network extracts high-level visual information through a deep neural network ; The visual student network extracts the full information of the visual modality through a shallow neural network Similar information to visual cross-modality Then perform subtraction to obtain visual modality specific information ;Will Input to classifier 2 , and get the second vector ,based on and The structural loss of ;

[0011] Step 4: Use the attention mechanism to and Perform weighted fusion to obtain a multimodal fusion vector ,Will After input to classifier three Get the third vector ,based on and Structural loss three ;Will and After classifier four And classifier five , and get the fourth vector and the fifth vector ;right and Constrain so that and The distance between and Constrain so that and The distance increases; based on Defining Loss VI ;Will After passing through the multi-layer perceptron, we can obtain the cross-modal similarity information of visual transformation , text transformation cross-modal similarity information , text transformation full information , visual transformation full information , text transformation specific information , visual transformation specific information ,based on Defining Loss 7 ;based on and The distance and and Distance construction loss 4 ,based on and The distance and and The distance structure loss is five ;

[0012] Step 5: and Input into the basic recommendation model to obtain the user's Prediction score results of items ,based on and users The actual rating labels of items The cross entropy loss constructs the loss of the basic recommendation model ;based on Total structural loss ;pass The basic recommendation model, text teacher network, text student network, visual teacher network, visual student network, classifiers one to five, and multi-layer perceptron are trained; the visual student network, text student network and basic recommendation model constitute the multimodal recommendation model that has completed the training.

[0013] Furthermore, step 1 specifically includes: forming a set of users in the recommendation data set of users interacting with items , record the set of all items as the set , the interaction matrix between users and items is recorded as ; If the set The User pair set The If an item produces positive behavior, then ; If User to If an item does not produce positive behavior, then ;

[0014] The text features corresponding to the text information of the items are extracted through the pre-trained Bert model ; Extract the visual features corresponding to the image information of the items through the pre-trained VGG19 model ; The text information of the item includes the user's comments on the item and the description of the item;

[0015] Perform one-hot encoding on the semantic labels of the items to obtain the semantic label vector of the items. The semantic label vector of an item is recorded as .

[0016] Furthermore, in step 2, the text features The text teacher network and the text student network are respectively passed through; the text teacher network extracts high-level information of the text through a deep neural network , the text student network extracts the full information of the text modality through a shallow neural network Similar information to text cross-modality Then perform subtraction to obtain text modality specific information , specifically including:

[0017] Text Teacher Network extracts text features through deep neural network High-level information of text :

[0018] ;

[0019] Represents the deep neural network in the text teacher network;

[0020] The text student network includes a first integrated mapping module and a second integrated mapping module; the first integrated mapping module and the second integrated mapping module both include A shallow neural network;

[0021] Text features are respectively input into the first integrated mapping module A shallow neural network, Text modal information, For the Text modal information; Aggregate the text modal information to obtain the full text modal information :

[0022] ;

[0023] ;

[0024] The first integrated mapping module A shallow neural network;

[0025] Text features are respectively input into the second integrated mapping module A shallow neural network, text cross-modal similarity information components, For the text cross-modal similarity information components; The text cross-modal similarity information components are aggregated to obtain the text cross-modal similarity information :

[0026] ;

[0027] ;

[0028] The second integrated mapping module A shallow neural network;

[0029] right and Do the difference and get the text modality specific information :

[0030] .

[0031] Further, in step 2, the Input to classifier 1 , get the first vector ,based on and The structural loss of , specifically including:

[0032] Classifier 1 Including neural networks and softmax functions;

[0033] High-level information of the text After a layer of neural network transformation, and then through the softmax function, we get :

[0034] ;

[0035] represents the temperature parameter, Respectively represent the weight parameter and bias parameter of classifier one, represents the softmax function;

[0036] based on and the semantic label vector of the item Calculate loss 1 :

[0037] ;

[0038] represents the cross entropy loss, Represents the semantic label vector The The value of the dimension, Represents the semantic label vector The total dimension of .

[0039] Further, in step 3, the visual feature After the visual teacher network and the visual student network; the visual teacher network extracts high-level visual information through a deep neural network ; The visual student network extracts the full information of the visual modality through a shallow neural network Similar information to visual cross-modality Then perform subtraction to obtain visual modality specific information , specifically including:

[0040] Visual Teacher Network extracts visual features via deep neural network Visual high-level information :

[0041] ;

[0042] Represents the deep neural network in the visual teacher network;

[0043] The visual student network includes a third integrated mapping module and a fourth integrated mapping module; the third integrated mapping module and the fourth integrated mapping module both include A shallow neural network;

[0044] The visual features are input into the third integrated mapping module A shallow neural network, Visual modality information, For the Visual modality information; The visual modality information is aggregated to obtain the full visual modality information :

[0045] ;

[0046] ;

[0047] The third integrated mapping module A shallow neural network;

[0048] The visual features are input into the fourth integrated mapping module A shallow neural network, visual cross-modal similarity information components, For the visual cross-modal similarity information components; The visual cross-modal similarity information components are aggregated to obtain the visual cross-modal similarity information :

[0049] ;

[0050] ;

[0051] The fourth integrated mapping module A shallow neural network;

[0052] right and Do the difference and get the visual modality specific information :

[0053] .

[0054] Further, in step three, the Input to classifier 2 , and get the second vector ,based on and The structural loss of , specifically including:

[0055] Classifier 2 Including neural networks and softmax functions;

[0056] Visual high-level information After a layer of neural network transformation and softmax function, we get :

[0057] ;

[0058] represents the temperature parameter, represents the weight parameters and bias parameters of classifier 2, represents the softmax function;

[0059] based on and the semantic label vector of the item Calculate loss 2 :

[0060] ;

[0061] represents the cross entropy loss, Represents the semantic label vector The The value of the dimension, Represents the semantic label vector The total dimension of .

[0062] Furthermore, in step 4, the attention mechanism is used to and Perform weighted fusion to obtain a multimodal fusion vector ,Will After input to classifier three Get the third vector ,based on and Structural loss three , specifically including:

[0063] High-level information of the text and visual high-level information , through an attention mechanism, the weight is obtained :

[0064]

[0065] represents the activation function, Represent the learnable weight parameters and bias parameters respectively;

[0066] By weight Will and Fusion, get the multimodal fusion vector :

[0067] ;

[0068] Will After classifier three After the neural network transformation in , it is input into the softmax function and obtained ;

[0069] ;

[0070] represents the temperature parameter, They represent the weight parameters and bias parameters of classifier three respectively, represents the softmax function;

[0071] based on and the semantic label vector of the item Calculate loss three :

[0072] ;

[0073] represents the cross entropy loss.

[0074] Further, in step 4, the and After classifier four And classifier five , and get the fourth vector and the fifth vector ,right and Constrain so that and The distance is reduced, and Constrain so that and The distance increases, based on Defining Loss VI , specifically including:

[0075] Full text modal information and full information of visual modality Input to classifier four And classifier five ,get and :

[0076]

[0077]

[0078] Through contrastive learning, similar visual cross-modal information Similar information to text cross-modality To constrain and The distance is reduced; for visual modality specific information and text modal specific information To constrain and The distance increases; through Defining Loss VI :

[0079] ;

[0080] Represents the norm of a vector.

[0081] Further, in step 4, the After passing through the multi-layer perceptron, we can obtain the cross-modal similarity information of visual transformation , text transformation cross-modal similarity information , text transformation full information , visual transformation full information , text transformation specific information , visual transformation specific information ,based on Defining Loss 7 ;based on and The distance and and Distance construction loss 4 ,based on and The distance and and The distance structure loss is five , specifically including:

[0082] Will After the transformation of the multi-layer perceptron, we get ;Will After the transformation of the multi-layer perceptron, we get ; Through comparative learning, we get and The distance and and The sum of the distances :

[0083] ;

[0084] represents the norm of a vector;

[0085] Will After multi-layer perceptron The transformation of :

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] , They represent the learnable parameters of the multilayer perceptron respectively;

[0091] calculate and The distance and and The sum of the distances :

[0092] ;

[0093] calculate and The distance and and The sum of the distances :

[0094] ;

[0095] pass Defining Loss 7 :

[0096] ;

[0097] represents the activation function;

[0098] Loss 4 for and The distance and and The sum of the distances:

[0099] ;

[0100] Loss Five for and The distance and and The sum of the distances:

[0101] .

[0102] Furthermore, the total loss for:

[0103] ;

[0104] in, are all weight hyperparameters.

[0105] Compared with the prior art, the beneficial technical effects of the present invention are:

[0106] The present invention designs a teacher-student network, in which the teacher network uses rich semantic information for supervised learning, thereby capturing multimodal data more comprehensively; through knowledge distillation, this information is smoothly transferred to the student network, effectively solving the data sparsity problem in the recommendation system.

[0107] The present invention designs a set of feature extractors, which improves the effectiveness of feature extraction and ensures the directness and simplicity of the implementation process.

[0108] In-depth tests on actual data sets have proved that the method in this invention has better performance than the existing technology and has good application prospects in the field of multimodal recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 This is a schematic diagram of the principle structure of the present invention.

[0110] Figure 2 It is a schematic diagram of the flow chart of the actual reasoning of the present invention. DETAILED DESCRIPTION

[0111] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0112] like Figure 1As shown, the present invention adopts a teacher network and a student network in the information extraction process. The teacher network performs supervised learning through the semantic labels of the objects, and the student network extracts the full information and similar information of the text, as well as the full information and similar information of the vision through the decoupled modalities. By contrasting the constraints of learning, the information representations of different modalities are aligned. The information in the teacher network is transferred to the student network through the form of feature distillation and the form of target distillation. Finally, multiple loss functions are weighted and added to obtain the final optimization target.

[0113] Each classifier in the present invention includes a layer of neural network and a softmax function.

[0114] The present invention specifically comprises the following steps:

[0115] Step 1: Group each user in the recommendation dataset where users interact with items into a set , record the set of all items as the set , the interaction matrix between users and items is recorded as , if the set The User pair set The If an item generates positive behavior, such as clicks, purchases, etc., then ,otherwise, . For the The present invention extracts the text features corresponding to the text information of the items through the pre-trained Bert model. , extract the visual features corresponding to the image information of the item through the pre-trained VGG19 model ; Among them, the text information of the item includes the user's comments on the item and the description of the item. Respectively represent the dimensions The set of real field vectors .

[0116] No. The semantic label vector of an item after one-hot encoding is recorded as . Indicates The total number of semantic labels for items, also represented by the semantic label vector The total dimension of .

[0117] Step 1 specifically includes:

[0118] Perform one-hot encoding on the semantic label information of all items to obtain the encoded semantic label vector corresponding to each item. The semantic label vector of an item is recorded as .

[0119] Extract the first Text information of items, get text features .

[0120] Extract the first Image information of each item, obtaining visual features .

[0121] Step 2: Text features Through the text teacher network, the high-level information of the text is extracted through the deep neural network (DNN) In the text student network, the text modal information and text cross-modal similarity information components are extracted through an integrated shallow neural network, which are respectively denoted as , By The text modality information and the text cross-modal similarity information components are aggregated separately to obtain the full text modality information Similar information to text cross-modality , and then perform subtraction to obtain text modality specific information .

[0122] Step 2 specifically includes:

[0123] Text teacher network, extracting high-level information from text through deep neural network , Representation of the deep neural network in the text teacher network.

[0124] The text student network extracts the text modal information and the text cross-modal similarity information components through an integrated shallow neural network, which are respectively denoted as , . Represents the number of shallow neural networks in the ensemble.

[0125] ;

[0126] ;

[0127] right Aggregate separately to get the full information of text mode Similar information to text cross-modality :

[0128] ;

[0129] ;

[0130] right and Do the difference and get the text modality specific information :

[0131] .

[0132] Step 3: Visual features Through the visual teacher network, the high-level visual information is extracted through the deep neural network (DNN) In the visual student network, the visual modality information and visual cross-modal similarity information components are extracted through an integrated shallow neural network, which are respectively denoted as , .Will The visual modality information and visual cross-modal similarity information components are aggregated separately to obtain the full visual modality information Similar information to visual cross-modality , and then perform subtraction to obtain visual modality specific information .

[0133] Step three specifically includes:

[0134] The visual teacher network extracts high-level visual information through a deep neural network. , Representation of the deep neural network in the visual teacher network.

[0135] The visual student network extracts the visual modality information and visual cross-modal similarity information components through an integrated shallow neural network, which are denoted as , . Represents the number of shallow neural networks in the vision student network ensemble.

[0136] ;

[0137] ;

[0138] right Aggregate separately to get the full information of visual modality Similar information to visual cross-modality :

[0139] ;

[0140] ;

[0141] right and Do the difference and get the visual modality specific information :

[0142] .

[0143] Step 4: High-level information of the text , after being transformed by a layer of neural network and then by the softmax function, we get ,based on and semantic label vector Calculate loss 1 , specifically including:

[0144] ;

[0145] ;

[0146] here, Represents the semantic label vector The The value of the dimension.

[0147] Step 5: High-level visual information , after being transformed by a layer of neural network and then by the softmax function, we get ,based on and semantic label vector Structural loss 2 , specifically including:

[0148] ;

[0149] .

[0150] Step 6: High-level information in the text and visual high-level information , the weight obtained through an attention mechanism, and and Weighted to obtain the weighted multimodal fusion vector , after a layer of neural network transformation, and then through the softmax function, we get ,based on and semantic label vector Structural loss three .

[0151] Step 6 specifically includes:

[0152] High-level information of the text and visual high-level information Through the attention mechanism, we get the weight :

[0153] ;

[0154] By weight ,Will and Combine them to get the multimodal fusion vector :

[0155] .

[0156] After a layer of neural network transformation, and then through the softmax function, we get ,based on and semantic label vector Calculate the cross entropy loss function :

[0157] ;

[0158] ;

[0159] represents the cross entropy loss.

[0160] Step 7: Full text modal information and full information of visual modality After being transformed by a layer of neural network and then by the softmax function, we get and Through the contrastive learning module, visual cross-modal similarity information is Similar information to text cross-modality Constrain so that and The distance is reduced, and the visual modality specific information and text modal specific information Constrain so that and The distance increases through Defining Loss VI .

[0161] Step seven specifically includes the following:

[0162] Full text modal information and full information of visual modality After classifier four And classifier five , each classifier includes a layer of neural network and softmax function, and we get and :

[0163] ;

[0164] ;

[0165] Through contrastive learning module, visual cross-modal similarity information Similar information to text cross-modality Constrain so that and Close distance, specific information for visual modality and text modal specific information Constrain so that and The distance is far. Define the loss function :

[0166] .

[0167] Step 8: and After passing through the multi-layer perceptron, we can obtain the cross-modal similarity information of visual transformation Transform similar information across modalities with text ; Through comparative learning, we get and The distance and and The sum of the distances ;Will After passing through the multi-layer perceptron, the full information of text transformation is obtained , visual transformation full information , text transformation specific information , visual transformation specific information ,calculate and The distance and and The sum of the distances ,calculate and The distance and and The sum of the distances ,pass Defining Loss 7 .

[0168] Step 8 specifically includes:

[0169] Will After the transformation of the multi-layer perceptron (MLP), we get ,Will After the transformation of the multi-layer perceptron (MLP), we get By comparing the learning modules, we can get and The distance and and The sum of the distances :

[0170] .

[0171] Will After being transformed by the multi-layer perceptron (MLP), we get :

[0172] ;

[0173] ;

[0174] ;

[0175] .

[0176] calculate and The distance and and The sum of the distances :

[0177] ;

[0178] calculate and The distance and and The sum of the distances :

[0179] .

[0180] pass Defining Loss 7 :

[0181] .

[0182] Step 9: Through knowledge distillation, define two constraints, including feature-based distillation and target-based distillation, including:

[0183] Feature-based Distillation: Computation and The distance and and The sum of the distances :

[0184] ;

[0185] Goal-based distillation: Computation and The distance and and The sum of the distances :

[0186] ;

[0187] Step 10: Define total loss , specifically including:

[0188] ;

[0189] in, Represents the weight of different losses and is a hyperparameter. and Input into the basic recommendation model to obtain the user's Prediction score results of items , It is based on the user's Prediction score results of items and users The actual rating labels of items The loss of the basic recommendation model constructed by the cross entropy loss.

[0190] When training the multimodal recommendation model, the following parameters are used:

[0191] ;

[0192] The number of training rounds is 1000, the number of extraction termination rounds is 20, the learning rate is 0.0001, and the optimizer is the Adam optimizer.

[0193] During actual reasoning, the text modality information output by the text student network is , the visual modality full information output by the visual student network , as well as user-side features (such as the user's serial number), item-side features (such as the item's serial number), context features and other information are input into the basic recommendation model to obtain the user's score on the item, and to recommend items to specific users. Among them, user-side features, item-side features, and context features are optional information that can be input into the basic recommendation model.

[0194] The basic recommendation model may adopt a common recommendation model in the existing technology such as Deep Factorization Machine (DeepFM).

[0195] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0196] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A knowledge distillation multimodal recommendation method based on modal decoupling, characterized in that: Recommend items to specific users through the trained multimodal recommendation model; the training process of the multimodal recommendation model includes: Step 1: Extract the text features corresponding to the text information of the item , extract the visual features corresponding to the image information of the item ;No. The semantic label vector of an item is ; Step 2: Text features The text teacher network and the text student network are respectively passed through; the text teacher network extracts high-level information of the text through a deep neural network ; The text student network extracts the full information of the text modality through a shallow neural network Similar information to text cross-modality Then perform subtraction to obtain text modality specific information ;Will Input to classifier 1 , get the first vector ,based on and The structural loss of ; Step 3: Visual features After the visual teacher network and the visual student network; the visual teacher network extracts high-level visual information through a deep neural network ; The visual student network extracts the full information of the visual modality through a shallow neural network Similar information to visual cross-modality Then perform subtraction to obtain visual modality specific information ;Will Input to classifier 2 , and get the second vector ,based on and The structural loss of ; Step 4: Use the attention mechanism to and Perform weighted fusion to obtain a multimodal fusion vector ,Will After input to classifier three Get the third vector ,based on and Structural loss three ;Will and After classifier four And classifier five , and get the fourth vector and the fifth vector ;right and Constrain so that and The distance between and Constrain so that and The distance increases; based on Defining Loss VI ;Will After passing through the multi-layer perceptron, we can obtain the cross-modal similarity information of visual transformation , text transformation cross-modal similarity information , text transformation full information , visual transformation full information , text transformation specific information , visual transformation specific information ,based on Defining Loss 7 ;based on and The distance and and Distance construction loss 4 ,based on and The distance and and The distance structure loss is five ; Step 5: and Input into the basic recommendation model to obtain the user's Prediction score results of items ,based on and users The actual rating labels of items The cross entropy loss constructs the loss of the basic recommendation model ;based on Total structural loss ;pass The basic recommendation model, text teacher network, text student network, visual teacher network, visual student network, classifiers one to five, and multi-layer perceptron are trained; the visual student network, text student network and basic recommendation model constitute the multimodal recommendation model that has completed the training.

2. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1 is characterized in that: Step 1 specifically includes: forming a set of users in the recommendation data set of users and items interacting , record the set of all items as the set , the interaction matrix between users and items is recorded as ; If the set The User pair set The If an item produces positive behavior, then ; If User to If an item does not produce positive behavior, then ; The text features corresponding to the text information of the items are extracted through the pre-trained Bert model ; Extract the visual features corresponding to the image information of the items through the pre-trained VGG19 model ; The text information of the item includes the user's comments on the item and the description of the item; Perform one-hot encoding on the semantic labels of the items to obtain the semantic label vector of the items. The semantic label vector of an item is recorded as .

3. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 2, the text features The text teacher network and the text student network are respectively passed through; the text teacher network extracts high-level information of the text through a deep neural network , the text student network extracts the full information of the text modality through a shallow neural network Similar information to text cross-modality Then perform subtraction to obtain text modality specific information , specifically including: Text Teacher Network extracts text features through deep neural network High-level information of text : ； Represents the deep neural network in the text teacher network; The text student network includes a first integrated mapping module and a second integrated mapping module; the first integrated mapping module and the second integrated mapping module both include A shallow neural network; Text features are respectively input into the first integrated mapping module A shallow neural network, Text modal information, For the Text modal information; Aggregate the text modal information to obtain the full text modal information : ； ； The first integrated mapping module A shallow neural network; Text features are input into the second integrated mapping module A shallow neural network, text cross-modal similarity information components, For the text cross-modal similarity information components; The text cross-modal similarity information components are aggregated to obtain the text cross-modal similarity information : ； ； The second integrated mapping module A shallow neural network; right and Do the difference and get the text modality specific information : 。 4. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 2, the Input to classifier 1 , get the first vector ,based on and The structural loss of , specifically including: Classifier 1 Including neural networks and softmax functions; High-level information of the text After a layer of neural network transformation, and then through the softmax function, we get : ； represents the temperature parameter, Respectively represent the weight parameters and bias parameters of classifier one, represents the softmax function; based on and the semantic label vector of the item Calculate loss 1 : ； represents the cross entropy loss, Represents the semantic label vector The The value of the dimension, Represents the semantic label vector The total dimension of .

5. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 3, the visual features After the visual teacher network and the visual student network; the visual teacher network extracts high-level visual information through a deep neural network ; The visual student network extracts the full information of the visual modality through a shallow neural network Similar information to visual cross-modality Then perform subtraction to obtain visual modality specific information , specifically including: Visual Teacher Network extracts visual features via deep neural network Visual high-level information : ； Represents the deep neural network in the visual teacher network; The visual student network includes a third integrated mapping module and a fourth integrated mapping module; the third integrated mapping module and the fourth integrated mapping module both include A shallow neural network; The visual features are input into the third integrated mapping module A shallow neural network, Visual modality information, For the Visual modality information; The visual modality information is aggregated to obtain the full visual modality information : ； ； The third integrated mapping module A shallow neural network; The visual features are input into the fourth integrated mapping module A shallow neural network, visual cross-modal similarity information components, For the visual cross-modal similarity information components; The visual cross-modal similarity information components are aggregated to obtain the visual cross-modal similarity information : ； ； The fourth integrated mapping module A shallow neural network; right and Do the difference and get the visual modality specific information : 。 6. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step three, the Input to classifier 2 , and get the second vector ,based on and The structural loss of , specifically including: Classifier 2 Including neural networks and softmax functions; Visual high-level information After a layer of neural network transformation and softmax function, we get : ； represents the temperature parameter, represents the weight parameters and bias parameters of classifier 2, represents the softmax function; based on and the semantic label vector of the item Calculate loss 2 : ； represents the cross entropy loss, Represents the semantic label vector The The value of the dimension, Represents the semantic label vector The total dimension of .

7. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 4, the attention mechanism is used to and Perform weighted fusion to obtain a multimodal fusion vector ,Will After input to classifier three Get the third vector ,based on and Structural loss three , specifically including: High-level information of the text and visual high-level information , through an attention mechanism, the weight is obtained : represents the activation function, Represent the learnable weight parameters and bias parameters respectively; By weight Will and Fusion, get the multimodal fusion vector : ； Will After classifier three After the neural network transformation in , it is input into the softmax function and obtained ; ； represents the temperature parameter, They represent the weight parameters and bias parameters of classifier three respectively, represents the softmax function; based on and the semantic label vector of the item Calculate loss three : ； represents the cross entropy loss.

8. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 4, the and After classifier four And classifier five , and get the fourth vector and the fifth vector ,right and Constrain so that and The distance is reduced, and Constrain so that and The distance increases, based on Defining Loss VI , specifically including: Full text modal information and full information of visual modality Input to classifier four And classifier five ,get and : Through contrastive learning, similar visual cross-modal information Similar information to text cross-modality To constrain and The distance is reduced; for visual modality specific information and text modal specific information To constrain and The distance increases; through Defining Loss VI : ； Represents the norm of a vector.

9. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: In step 4, the After passing through the multi-layer perceptron, we can obtain the cross-modal similarity information of visual transformation , text transformation cross-modal similarity information , text transformation full information , visual transformation full information , text transformation specific information , visual transformation specific information ,based on Defining Loss 7 ;based on and The distance and and Distance construction loss 4 ,based on and The distance and and The distance structure loss is five , specifically including: Will After the transformation of the multi-layer perceptron, we get ;Will After the transformation of the multi-layer perceptron, we get ; Through comparative learning, we get and The distance and and The sum of the distances : ； represents the norm of a vector; Will After multi-layer perceptron The transformation of : ； ； ； ； , They represent the learnable parameters of the multilayer perceptron respectively; calculate and The distance and and The sum of the distances : ； calculate and The distance and and The sum of the distances : ； pass Defining Loss 7 : ； represents the activation function; Loss 4 for and The distance and and The sum of the distances: ； Loss Five for and The distance and and The sum of the distances: 。 10. The knowledge distillation multimodal recommendation method based on modal decoupling according to claim 1, characterized in that: The total loss for: ； in, are all weight hyperparameters.

Images