Method for performing robust multimedia recommendation on basis of information bottleneck, and device and medium

Abstract

A method for performing robust multimedia recommendation on the basis of an information bottleneck, and a device and a medium. The method comprises: on the basis of a deep graph neural model, learning a user representation matrix and a product representation matrix; on the basis of an information bottleneck theory, minimizing mutual information between multimedia content and representation information of the multimedia content, and calculating a first loss function; reconstructing an interaction matrix on the basis of the user representation matrix and the product representation matrix, and calculating a second loss function; and performing multi-task learning by combining the loss functions, so as to update model parameters until a model converges.

Description

Robust multimedia recommendation method, device and medium based on information bottleneck

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311351239.9, filed on October 18, 2023, entitled “A Multimedia Recommendation Method Based on Information Bottleneck Denoising,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of multimedia recommendation technology, and in particular to a robust multimedia recommendation method, device and medium based on information bottleneck. Background Art

[0004] Recommender systems are information technologies that provide personalized services and are often used to alleviate information overload caused by massive amounts of data. Traditional ID-based collaborative filtering methods use techniques such as matrix decomposition and graph neural networks to characterize user preferences and achieve personalized product recommendations. With the development of deep learning and multimedia information processing technologies, recommendation methods based on multimedia data have been widely researched.

[0005] In related technologies, multimedia recommendation systems can use pre-trained models to extract features from multimedia content and then combine them with ID information to model user preferences. However, because pre-trained models are typically task-oriented, such as visual pre-trained models trained on visual tasks, the extracted visual features are noisy for recommendation tasks, resulting in poor user preference modeling and less than ideal multimedia recommendation accuracy.

[0006] Summary of the Invention

[0007] In response to the deficiencies of the prior art, the present application provides a robust multimedia recommendation method, device, and medium based on information bottleneck, which solves the problem of inaccurate multimedia recommendations due to the presence of characteristic noise.

[0008] To achieve the above objectives, this application is implemented through the following technical solutions:

[0009] In the first aspect, an embodiment of the present application provides a robust multimedia recommendation method based on information bottleneck, which includes: constructing heterogeneous data: a user-product interaction matrix, and multimedia content data of the product; using a pre-trained model to extract multimedia content features of the product; constructing a product association matrix based on the multimedia content features; learning user and product representation matrices based on a deep graph neural model; minimizing the mutual information between multimedia content and its representation information based on the information bottleneck theory, and calculating a first loss function; reconstructing an interaction matrix based on the user representation matrix and the product representation matrix, and calculating a second loss function; and performing multi-task learning by combining various loss functions to update model parameters until the model converges.

[0010] In a second aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the robust multimedia recommendation method based on information bottleneck in the aforementioned first aspect is implemented.

[0011] In a third aspect, an embodiment of the present application provides a non-temporary readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the robust multimedia recommendation method based on information bottleneck in the first aspect is implemented.

[0012] This application provides a robust multimedia recommendation method, device, and medium based on information bottlenecks. Compared with existing technologies, it has the following advantages:

[0013] This application extracts the multimedia content features of products through a pre-trained model to construct a product association matrix, and learns the user and product representation matrices through a deep graph neural model without introducing any model parameters. The method is very lightweight and has strong practical value. It constructs information bottleneck constraints through the information bottleneck theory, minimizes the mutual information between multimedia content and its representation information, and obtains the minimum multimedia content that meets the recommendation task. The pre-trained multimedia features are then denoised, which can implicitly reduce the impact of multimedia noise on user modeling, thereby improving the quality of user preference modeling and improving recommendation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] FIG1 is a flow chart of a robust multimedia recommendation method based on information bottleneck provided by an embodiment of the present application;

[0016] FIG2 is another exemplary flowchart of the robust multimedia recommendation method based on information bottleneck provided in an embodiment of the present application;

[0017] FIG3 is a schematic diagram of an exemplary process of S110 in FIG1 ;

[0018] FIG4 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0021] The embodiments of the present application provide a robust multimedia recommendation method, device and medium based on information bottleneck, which solves the problem of inaccurate multimedia recommendations due to the presence of feature noise, realizes implicit denoising of pre-trained multimedia features, improves the quality of user preference modeling, and enhances recommendation performance.

[0022] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0023] Deep learning and graph neural networks are two important branches in the field of artificial intelligence. Deep learning has been widely used in fields such as computer vision and natural language processing, while graph neural networks focus on processing data with graph structures. The fusion of deep learning and graph neural networks is gradually being used to solve more complex technical problems.

[0024] Recommender systems are information technologies that provide personalized services and are often used to alleviate information overload caused by massive amounts of data. Traditional ID-based collaborative filtering methods characterize user preferences through techniques such as matrix decomposition and graph neural networks, enabling personalized product recommendations. Currently, with the advancement of deep learning and multimedia information processing technologies, existing multimedia recommendation methods can use pre-trained models to extract features from multimedia content. These models then combine ID information to model user preferences. Once this modeling is complete, targeted multimedia information product recommendations can be made to users.

[0025] However, in related technologies, pre-trained models are usually task-oriented. For example, visual pre-training models are trained based on visual tasks. The visual feature information extracted by the visual pre-training model contains noise information for recommendation tasks and needs to be denoised. Multimedia recommendation methods generally have the problem of feature noise. Since feature noise is difficult to formally define, how to implicitly design a denoising module to improve recommendation tasks is a key issue.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] The following first introduces a robust multimedia recommendation method based on information bottleneck provided by an embodiment of the present application.

[0028] An embodiment of the present application provides a flowchart of a robust multimedia recommendation method based on information bottleneck. Please refer to FIG. 1 and FIG. 2 . The robust multimedia recommendation method based on information bottleneck may include the following steps S110 - S170 .

[0029] S110. Construct heterogeneous data: user interaction matrix with the product, and product multimedia content data.

[0030] S120: Extract multimedia content features of the product using a pre-trained model.

[0031] S130: Construct a product association matrix based on multimedia content features.

[0032] S140. Learning user and product representation matrices based on deep graph neural models.

[0033] S150 , minimizing the mutual information between the multimedia content and its representation information based on the information bottleneck theory, and calculating a first loss function.

[0034] S160: Reconstruct the interaction matrix based on the user representation matrix and the product representation matrix, and calculate the second loss function.

[0035] S170: Combine the loss functions to perform multi-task learning to update the model parameters until the model converges.

[0036] In the embodiments of the present application, it should be noted that deep learning is a machine learning method based on neural networks, which simulates the working principle of the human brain through a multi-level neural network to achieve the representation and learning of complex data. The graph neural network is a neural network model specifically used to process graph data, which can capture the relationship between nodes and the global structure; the deep graph neural model is a deep graph neural network. The deep graph neural network is an organic combination of deep learning and graph neural networks. It aims to use the powerful representation ability of deep learning and the graph structure processing ability of the graph neural network to accurately and efficiently model and predict graph data. In a deep graph neural network, each node can be regarded as a sample, and the connection relationship between nodes can be represented as the edge of the graph.

[0037] It's also worth noting that the information bottleneck theory is an extension of the rate-distortion theory of source compression. In traditional data compression, due to the lack of prior knowledge, the input data must be memorized, but redundancy in the data itself can be eliminated. If data compression allows for distortion, rate-distortion theory allows for a trade-off between distortion and compression rate. Based on the information bottleneck theory, if the data has prior knowledge, such as labeled data, in addition to eliminating redundancy in the data itself, information unrelated to the label can be compressed and forgotten, further eliminating redundant information irrelevant to the learning task. This allows for efficient compression while retaining label-related information.

[0038] It can be understood that after constructing heterogeneous data and determining the user's interaction matrix with the product, as well as information such as the product's multimedia content data, this application can use a pre-trained model to extract multimedia content features and perform matrix analysis on the extracted multimedia content features to construct a product association matrix. Based on the product association matrix, the product association matrix can be further processed and participate in the processing process of the aforementioned deep graph neural model learning the user and product representation matrix.

[0039] It should be emphasized that the user and product representation matrix includes the user representation matrix and the product representation matrix; the deep graph neural model is the aforementioned deep graph neural network; the first loss function determined in this application minimizes the mutual information between multimedia content and its representation information based on the information bottleneck theory, and efficiently compresses and extracts the mutual information, forgetting irrelevant redundant information, thereby ensuring the accuracy of data learning; this application also fuses the user representation matrix and the product representation matrix, reconstructs the interaction matrix, and calculates and determines the second loss function to combine the aforementioned first loss function and the second loss function for multi-task learning, and updates the aforementioned deep graph neural model to convergence; based on this, this application can effectively remove feature noise and achieve accurate multimedia recommendations.

[0040] The above is a specific implementation method of the robust multimedia recommendation method based on information bottleneck provided in the embodiment of the present application. The present application learns the user and product representation matrices through a deep graph neural model without introducing any model parameters, and the method usage process is very lightweight; the information bottleneck constraint is constructed through the information bottleneck theory, and the mutual information between multimedia content and its representation information is minimized to obtain the minimum multimedia content that meets the recommendation task, and then the pre-trained multimedia features are denoised, which can implicitly reduce the impact of multimedia noise on user modeling, thereby improving the quality of user preference modeling and improving recommendation performance.

[0041] In some embodiments, the information bottleneck-based robust multimedia recommendation method may further include: Step 1.1, let U represent the user set, and U = {u1, ...u a ,...,u b ,...u M},u a represents the ath user, u b represents the b-th user, M represents the total number of users, 1≤a, b≤M.

[0042] Let V denote the product set, and V = {v1, ..., v i ,...,v j ,...,v N}, v i represents the i-th product, v j represents the j-th product, N represents the total number of products, 1≤i, j≤N.

[0043] Let r ai represents the a-th user u a For the i-th product v i The interaction data of users on products, then the user interaction matrix R = {r ai} M×N .

[0044] Step 1.2, let C = {C 1 ,...,C k ,...,C K} represents the multimedia content of the product set, where C k represents the multimedia content of the kth modality in the product set V, and represents the multimedia content of the kth modality of the product set V, represents the i-th product v i The multimedia content of the kth modality, K represents the total number of modalities, 1≤k≤K.

[0045] Use the pre-trained model of each modality to extract features of the corresponding modality multimedia content, and obtain the multimedia feature matrix set of K modalities {F 1 ,...,F k ,...,F K}, where F k Represents the multimedia feature matrix of the kth modality.

[0046] Step 3.1: Calculate the similarity matrix set {S 1 ,...,S k ,...,S K}. S k =L2(F k )×(L2(F k )) T (1)

[0047] In formula (1), S k Represents the similarity matrix of the kth mode in the product set V, L2(F k ) represents the normalized multimedia feature matrix of the kth modality in the product set V, and T represents the matrix transpose.

[0048] Step 3.2: According to formula (2), the similarity matrix set {S 1 ,...,S k ,...,S K} is sparsely processed to obtain the correlation matrix set {S′ 1 ,...,S′ k ,...,S′ K}.

[0049] In formula (2), Represents the similarity matrix S of the kth mode in the product set V k The element value of the i-th row and j-th column of Represents the correlation matrix S′ of the kth mode in the product set V k The element value of the i-th row and j-th column of Represents the similarity matrix S of the kth mode in the product set V k All element values ​​in the i-th row of , TopK′ represents the first K′ in descending order.

[0050] Step 3.3: Calculate the correlation matrix S′ of the product set V according to formula (3):

[0051] Step 3.4: Calculate the representation propagation matrix A according to formula (4):

[0052] The normalized representation propagation matrix is ​​calculated by formula (5):

[0053] In formula (5), D represents the degree matrix of matrix A.

[0054] Step 4.1: Use Gaussian distribution to randomly initialize the user representation matrix P = {p1, ..., p a ,...,p M}, where p a represents the a-th user u a The d1+Kd2-dimensional representation vector of , where d1 represents the user collaborative representation dimension and d2 represents the content representation dimension.

[0055] Use Gaussian distribution to randomly initialize the product co-representation matrix Q = {q1, ..., q i ,...,q N}, where q i represents the i-th product v i The d1-dimensional representation vector of .

[0056] Step 4.2: Calculate the multimedia content representation matrix T of each modality in the product set V by formula (6) = {T 1 ,...,T k ,...,T K}: T k =MLP k (F k ) (6)

[0057] In formula (6), MLP k is the kth multilayer perceptron, Represents the multimedia content representation matrix of the kth modality.

[0058] Step 4.3: Construct the product representation matrix E = [Q, T] and use the graph neural network to process {P, E} to obtain the user final representation matrix H u And the final representation matrix H v .

[0059] Step 5.1, use PCA algorithm to analyze the multimedia feature matrix set {F 1 ,...,F k ,...,F K} to reduce the dimension and obtain the multimodal feature matrix {F′ 1 ,...,F′ k ,...,F′ K},in, represents the kth mode F k Multimedia feature matrix after dimensionality reduction.

[0060] Step 5.2: Construct the information bottleneck loss function L(θ1) according to formula (10):

[0061] In formula (10), θ1 represents the parameters corresponding to K multi-layer perceptrons, and HSIC(,) represents the Hilbert-Schmidt independence index. It should be noted that the information bottleneck loss function L(θ1) is the first loss function mentioned above.

[0062] Step 6.1: Predict the ath user u according to formula (11) a For the i-th product v i The interaction probability r′ ai :

[0063] In formula (11), δ is the sigmoid activation function; Represents the user's final representation matrix H u The ath row in represents the transpose of the vector; Represents the final product representation matrix H v The i-th row in represents the vector.

[0064] Step 6.2: Calculate the reconstruction loss function L(θ2) according to formula (12):

[0065] In formula (12), θ2 represents all parameters involved in reconstructing the interaction matrix, and θ2 = [P, Q, θ1]; D a is the ath user u a All training data of D a ={(i, j)|i∈R a , j∈VR a}; R arepresents the αth user u a A collection of interactive products; Represents the final product representation matrix H v It should be noted that the reconstruction loss function L(θ2) is the second loss function mentioned above.

[0066] Step 7.1. Establish the multi-task optimization objective L(θ) according to formula (13): L(θ) = L(θ1) + αL(θ2) + β||[P, Q,]|| 2 (13)

[0067] In formula (13), θ = [P, Q, θ2] is all the parameters to be optimized, and α and β are used to control the weights of the reconstruction loss function and the regularization term, respectively.

[0068] Step 7.2: Solve the optimization target L(θ) by gradient descent to update the parameter θ until L(θ) converges to the minimum value, thus obtaining the optimal parameter θ. * .

[0069] Step 7.3: Predict the ath user u according to formula (14) a For the i-th product v i The optimal interaction probability Thus we get the interaction matrix of user set U to product set V prediction Used to implement product recommendations:

[0070] In formula (14), represents the a-th user u a The optimal representation vector of represents the i-th product v i The optimal representation vector of .

[0071] In some embodiments, the graph neural network in step 4.3 has a total of L convolutional layers, the current convolutional layer is l, and l is initialized to 0;

[0072] Initialize the l-th layer node representation matrix H l ={P, E};

[0073] The l-th layer node representation matrix H l It is sent into the graph neural network, and the l+1 layer node representation matrix H is calculated according to formula (7) l+1

[0074] According to formula (8), the output of each graph convolution layer is aggregated to obtain the final node representation matrix H:

[0075] According to formula (9), the user's final representation matrix H is obtained u And the final representation matrix H v : H u =H[:M],H v =H[M:] (9)

[0076] In formula (9), H[:M] represents the 1st to Mth rows of the matrix H, and H[M:] represents the (M+1)th to (M+N)th rows of the matrix H.

[0077] In some embodiments, referring to FIG. 3 , the aforementioned process of constructing heterogeneous data, i.e., S110 , may specifically include the following steps:

[0078] S210: Acquire a product set and a user set, where the product set includes multimedia content data, and the user set includes a plurality of user data information corresponding to the multimedia content data.

[0079] S220 : Determine a first mapping relationship between multiple users in the user set and each sub-data in the multimedia content data, and obtain multiple first target data to represent the interaction relationship between the user set and the product set.

[0080] S230: Construct a user-product interaction matrix based on the plurality of first target data.

[0081] In the embodiment of the present application, it can be understood that in the process of recommending multimedia products to different users, it is necessary to obtain and analyze the previous product set and user set; wherein, the user data information and the multimedia content data correspond to each other, and by analyzing the interaction relationship between the user set and the product set, a user-product interaction matrix can be obtained.

[0082] In one example, in the process of analyzing the interaction relationship between the user set and the product set, the user set can be represented by U, and U={u1,...u a ,...,u b ,...u M},u a represents the ath user, u b represents the bth user, M represents the total number of users, 1≤a, b≤M; r ai represents the a-th user u a For the i-th product v i The interaction data of users on products, then the user interaction matrix R = {r ai ) M×N .

[0083] In some embodiments, the pre-trained model includes a plurality of preset sub-trained models; the aforementioned extraction of multimedia content features of the product using the pre-trained model, i.e., S120, may specifically include the following steps:

[0084] S310: Divide the multimedia content data into multiple specific modalities, and establish a second mapping relationship between the multiple specific modalities and the multiple sub-training models.

[0085] S320. Based on the second mapping relationship, extract multimedia content features corresponding to each specific modality in the product set through multiple sub-training models to obtain multiple multimedia feature matrices and a multimedia feature matrix set.

[0086] In the embodiments of the present application, it can be understood that there is more than a single specific modality in the multimedia content data, that is, there are multiple categories in the multimedia content data. In order to perform targeted feature extraction on the multimedia content data, multiple preset sub-training models correspond to multiple specific modalities. Based on the mapping relationship between the multiple sub-training models and the multiple specific modalities, the sub-training models can be used to extract the multimedia content features of the product set in a targeted manner; further, to facilitate data analysis, multiple multimedia feature matrices and a set of multimedia feature matrices can be determined based on the results extracted from the product set.

[0087] In some embodiments, the multimedia content data corresponds to multiple specific modalities; the aforementioned construction of the product association matrix based on the multimedia content features, i.e., S130, may specifically include the following steps:

[0088] S410: Analyze the similarities of multiple specific modalities, and determine multiple similarity matrices and a similarity matrix set corresponding to the multiple specific modalities.

[0089] S420 , performing sparse processing and sorting on the multiple similarity matrices to obtain a target sequence in which multiple elements are sequentially arranged.

[0090] S430: Select a predetermined number of larger elements from the target sequence to determine a product association matrix and an association matrix set between multiple products.

[0091] In the embodiments of the present application, it can be understood that different multimedia products are not completely opposite in content. In the process of classifying multimedia content data based on specific modalities, products under different specific modalities are not completely different, that is, there is a certain degree of similarity between multiple specific modalities, and the degree of similarity between two different specific modalities also varies; based on this, in the process of considering different specific modalities for data analysis to achieve multimedia recommendations, the accuracy of the recommendations can be further guaranteed by analyzing and studying the similarities between multiple specific modalities.

[0092] It should be noted that after obtaining multiple similarity matrices, a set of significant target sequences can be obtained through sparse processing and sorting. A predetermined number of larger elements are selected from the target sequences to determine a set of significant association matrices for subsequent analysis.

[0093] In some embodiments, after constructing the product association matrix according to multimedia content features, that is, after S130, the robust multimedia recommendation method based on information bottleneck may further include the following steps:

[0094] S131. Determine the user-product interaction matrix as a first matrix subset, and transpose the first matrix subset to obtain a second matrix subset.

[0095] S132. Determine the product association matrix as the third matrix subset.

[0096] S133. Construct a first characterization propagation matrix based on the first matrix subset, the second matrix subset, and the third matrix subset.

[0097] In the embodiment of the present application, it is understood that after determining the user-product interaction matrix based on the aforementioned S230 and determining the significant product association matrix based on the aforementioned S430, in order to preliminarily establish the connection between the aforementioned analysis results and the deep graph neural model, the previous user set and product set are combined with the graph neural network in the deep graph neural model. Matrix aggregation processing can be performed based on the aforementioned interaction matrix and product association matrix to fuse the association relationships between users and products, and the association relationships between multiple products. Specifically, a first representational propagation matrix can be constructed based on the interaction matrix and the product association matrix; it should also be noted that the product association matrix is ​​consistent with the association matrix in the aforementioned step 3.3.

[0098] In one example, the first characterization propagation matrix A can be expressed as:

[0099] Among them, R is the interaction matrix, R T is the transposed matrix corresponding to the interaction matrix, and S′ is the product association matrix.

[0100] In some embodiments, after constructing the first representation propagation matrix based on the first matrix subset, the second matrix subset, and the third matrix subset, that is, after S133, the robust multimedia recommendation method based on information bottleneck may further include the following steps:

[0101] S134. Calculate a first degree matrix representing the propagation matrix.

[0102] S135 , performing a matrix multiplication operation on the degree matrix and the first characterization propagation matrix to normalize the first characterization propagation matrix to obtain a second characterization propagation matrix.

[0103] In the embodiment of the present application, it can be understood that after determining the first characterization propagation matrix, in order to more accurately combine it with the graph neural network in the deep graph neural model, the degree matrix of the first characterization propagation matrix can be analyzed, and then the first characterization propagation matrix can be normalized based on the degree matrix to obtain the second characterization propagation matrix; wherein, the second characterization propagation matrix can participate in the iterative calculation of the graph neural network, so that the deep graph neural model is more in line with data such as prior user sets and product sets.

[0104] In one example, the first characterization propagation matrix is ​​consistent with the characterization propagation matrix A in the aforementioned step 3.4, and the second characterization propagation matrix is ​​consistent with the characterization propagation matrix in the aforementioned step 3.4. consistent.

[0105] In some embodiments, the user and product representation matrix includes a user representation matrix and a product representation matrix; the aforementioned learning of the user and product representation matrix based on the deep graph neural model, i.e., S140, may specifically include the following steps:

[0106] S510 : randomly generating an initialized first user characterization matrix and an initialized first product collaborative characterization matrix through Gaussian distribution.

[0107] S520: Calculate the multimedia content representation matrix of the product through a preset multi-layer perceptron.

[0108] S530: Fuse the first product collaborative representation matrix and the multimedia content representation matrix to determine a product representation matrix.

[0109] S540: Fuse the first user representation matrix and the product representation matrix to determine a 0th-layer node representation matrix.

[0110] S550: Input the 0th layer node representation matrix into the preset graph neural network, use the second representation propagation matrix as the iteration coefficient, and obtain the intermediate node representation matrix corresponding to multiple convolutional layers.

[0111] S560. Based on the number of convolutional layers in the graph neural network, multiple intermediate node representation matrices and the 0th layer node representation matrix are aggregated to obtain a final node representation matrix.

[0112] In the embodiment of the present application, it can be understood that the deep graph neural model is composed of multiple convolutional layers, pooling layers and fully connected layers. In the process of analyzing the representation characteristics of users and products based on the deep graph neural model, the 0th layer convolutional layer can be initialized based on the Gaussian distribution. Specifically, the first user representation matrix and the first product collaborative representation matrix can be randomly generated by the Gaussian distribution, and the multimedia content representation matrix of the product can be calculated by the multi-layer perceptron. Based on the first user representation matrix, the first product collaborative representation matrix and the multimedia content representation matrix, a fusion analysis is performed to determine the 0th layer node representation matrix.

[0113] In addition, based on the aforementioned second representation propagation matrix and the 0th layer node representation matrix, the graph neural network is adjusted to obtain the intermediate node representation matrix corresponding to multiple convolutional layers; and further based on the number of convolutional layers, multiple intermediate node representation matrices and the 0th layer node representation matrix, the final node representation matrix is ​​obtained through aggregation processing.

[0114] In one example, the node final representation matrix H satisfies the expression:

[0115] Where L is the number of convolutional layers, l represents the current convolutional layer, and H l Represents the node representation matrix of the lth layer.

[0116] In another example, the first user characterization matrix can be expressed as P = {p1, ..., p a ,...,p M}, where p a represents the a-th user u a d1+Kd2-dimensional representation vector; where d1 represents the user collaborative representation dimension, d2 represents the content representation dimension; the first product collaborative representation matrix can be expressed as Q={q1,...,q i ,...,q N}, where q i represents the i-th product v j The d1-dimensional representation vector of the multimedia content representation matrix can be expressed as T = [T 1 ,...,T k ,...,T K ], where T k =MLP k (F k ), MLP k is the kth multilayer perceptron, Represents the multimedia content representation matrix of the kth modality.

[0117] Specifically, the product representation matrix E = [Q, T], and the 0th-layer node representation matrix is ​​{P, E}. Based on this, the final node representation matrix obtained by aggregating multiple intermediate node representation matrices and the 0th-layer node representation matrix covers both user information and product information.

[0118] In some embodiments, after aggregating multiple intermediate node representation matrices and the 0th layer node representation matrix based on the number of convolutional layers in the graph neural network to obtain a final node representation matrix, that is, after the aforementioned S560, the robust multimedia recommendation method based on information bottleneck may further include the following steps:

[0119] S570: Analyze and extract each row vector in the final node representation matrix to determine a final user representation matrix and a final product representation matrix.

[0120] In the embodiment of the present application, it can be understood that since the final node representation matrix is ​​obtained by aggregating multiple matrices and covers user information and product information; different subsets of the final node representation matrix can correspond to different information, by extracting each row vector of the final node representation matrix, the user final representation matrix and the product final representation matrix can be obtained.

[0121] In one example, the user's final representation matrix H u Satisfaction: H u =H[:M]

[0122] Where H[:M] represents the 1st to Mth rows of the final node representation matrix;

[0123] The final product characterization matrix H v Satisfaction: H v =H[M:]

[0124] Where H[M:] represents the (M+1)th to (M+N)th rows of the final node representation matrix.

[0125] In some embodiments, the representation information of the multimedia content is a multimedia content representation matrix, and the first loss function is an information bottleneck loss function; the aforementioned step of minimizing the mutual information between the multimedia content and its representation information based on the information bottleneck theory and calculating the first loss function, i.e., the aforementioned S150, may specifically include the following steps:

[0126] S610 , using a PCA algorithm to perform dimensionality reduction processing on the multimedia feature matrix set to obtain a multimodal feature matrix after dimensionality reduction.

[0127] S620. Based on the information bottleneck theory, the Hilbert-Schmidt independence index between the multimodal feature matrix and the multimedia content representation matrix is ​​calculated to obtain the information bottleneck loss function.

[0128] In the embodiments of the present application, it can be understood that the PCA algorithm is a principal component analysis method. PCA transforms the original data into a set of linearly independent representations of each dimension through linear transformation, which can be used to extract the main feature components of the data; the high-dimensional multimedia feature matrix set is reduced in dimension by the PCA algorithm to obtain a multimodal feature matrix after dimensionality reduction; based on the information bottleneck theory, the Hilbert-Schmidt independence index is constructed to obtain the information bottleneck loss function, which can provide constraints for data analysis, forget and eliminate redundant information irrelevant to the learning task, thereby ensuring the accuracy of data learning.

[0129] In one example, the information bottleneck loss function satisfies the expression: L(θ1)=∑HSIC(F′ k , T k ) (16)

[0130] Where L(θ1) represents the information bottleneck loss function, θ1 represents the parameters corresponding to K multi-layer perceptrons, HSIC(,) represents the Hilbert-Schmidt independence index, F′ k Represents the multimodal feature matrix after dimensionality reduction, T k Represents the multimedia content representation matrix.

[0131] In some embodiments, the second loss function is a reconstruction loss function; the aforementioned reconstructing the interaction matrix based on the user representation matrix and the product representation matrix and calculating the second loss function, that is, the aforementioned S160, may specifically include the following steps:

[0132] S710 , processing the user final representation matrix and the product final representation matrix through a preset sigmoid activation function, respectively predicting the interaction probability of each user to the product, so as to reconstruct the user-product interaction matrix.

[0133] S720: Calculate a reconstruction loss function based on the predicted multiple interaction probabilities and all parameters involved in reconstructing the interaction matrix.

[0134] In the embodiment of the present application, the user's final representation matrix H is generated based on the depth map neural model analysis. u And the final representation matrix H v Afterwards, it can be understood that since the user finally represents the matrix H u And the final representation matrix H v The data range is large. To facilitate data analysis and statistics, the user's final representation matrix H can be transformed into u And the final representation matrix H vThe data in is mapped to [0, 1], and based on the user's final representation matrix H u And the final representation matrix H v The mapped data is used to predict the probability of interaction between users and products to reconstruct the interaction matrix predicted by the deep graph neural model and calculate the reconstruction loss function.

[0135] In one example, the aforementioned heterogeneous data also includes a product set; the aforementioned user interaction probability for a product satisfies the expression:

[0136] Where r′ ai is the interaction probability, δ is the sigmoid activation function, Represents the user's final representation matrix H u The ath row in represents the transpose of the vector; Represents the final product representation matrix H v The i-th row in represents the vector;

[0137] The reconstruction loss function L(θ2) satisfies the expression:

[0138] Where θ2 represents all parameters involved in reconstructing the interaction matrix and corresponds to the reconstruction loss function, θ2 = [P, Q, θ1], P is the first user representation matrix, Q is the first product collaborative representation matrix, and θ1 represents the parameters corresponding to K multi-layer perceptrons;

[0139] D a is the ath user u a All training data of D a ={(i, j)|i∈R a , j∈VR a}; R a represents the a-th user u a The interactive product set, V represents the product set; Represents the final product representation matrix H v The j-th row vector in .

[0140] In some embodiments, the first loss function is an information bottleneck loss function, and the second loss function is a reconstruction loss function; the aforementioned multi-task learning is performed by combining the respective loss functions to update the model parameters until the model converges, that is, the aforementioned S170 may specifically include the following steps:

[0141] S810. Establish a multi-task optimization objective function based on the information bottleneck loss function and the reconstruction loss function; wherein the multi-task optimization objective function corresponds to all parameters to be optimized.

[0142] S820. Solve the multi-task optimization objective function by using the gradient descent method to update all parameters to be optimized.

[0143] S830. Determine the parameter corresponding to when the multi-task optimization objective function converges to the minimum value, which is the optimal parameter after all parameters to be optimized are updated.

[0144] S840: Based on the optimal parameters, predict the target interaction probability of each user for the product to determine a target interaction matrix, which is used to implement product recommendations.

[0145] In the embodiments of the present application, it can be understood that the gradient descent method is an iterative algorithm. After obtaining the information bottleneck loss function and the reconstruction loss function, the corresponding multi-task optimization objective function can be constructed based on all the parameters to be optimized, and the multi-task optimization objective function can be solved by the gradient descent method until the multi-task optimization objective function converges, and then the optimal parameters are determined to obtain the target interaction matrix to recommend products to users.

[0146] In one example, the multi-task optimization objective function satisfies the expression: L(θ)=L(θ1)+αL(θ2)+β||[P, Q,]|| 2 (19)

[0147] Where θ = [P, Q, θ2] is all the parameters to be optimized, P is the first user representation matrix, Q is the first product collaborative representation matrix, and θ2 represents all the parameters corresponding to the reconstruction loss function. is the regularization term, α and β are the weights for regulating the reconstruction loss function and the regularization term respectively;

[0148] The target interaction matrix satisfies:

[0149] in,

[0150] Where, represents the αth user u a The target representation vector, represents the i-th product v i The target representation vector.

[0151] In some embodiments, to verify the effectiveness of this method, this application selects three commonly used Amazon datasets for experiments: Clothing, Sports, and Baby. For each user, one product that the user interacted with is randomly selected as the test set, and the remaining interaction data is used as the training set. For each user-product interaction record, one product that the user has not interacted with is randomly sampled to form a triple for model training; in the test phase, the present invention selects all products that the user has not interacted with for sorting, and uses Recall@N and Normalized Discounted Cumulative Gain (NDCG@N) as evaluation criteria.

[0152] This paper selected five related methods for comparison: BPR-MF, LightGCN, VBPR, LightGCN-M, and LATTICE. Specifically, Tables 1 to 3 show the experimental results on the three datasets mentioned above.

[0153] Table 1: Comparison of recommendation results between the proposed method and the comparative method on the Clothing dataset

[0154] Table 2: Comparison of recommendation results between the proposed method and the comparative method on the Sports dataset

[0155] Table 3: Comparison of recommendation results between the proposed method and the comparative method on the Baby dataset

[0156] It can be seen from the above three tables that on the three data sets, the IBRec method proposed in this invention is significantly better than the five compared methods in terms of the four indicators of Recall@10, NDCG@10, Recall@20 and NDCG@20.

[0157] In some embodiments, the present application also provides an electronic device, a schematic structural diagram of the electronic device is shown in FIG4 .

[0158] The electronic device may include a processor 910 and a memory 920 storing computer program instructions.

[0159] Specifically, the processor 910 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0160] The memory 920 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 920 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 920 may include removable or non-removable (or fixed) media. Where appropriate, the memory 920 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 920 is a non-volatile solid-state memory.

[0161] The memory 920 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, generally, the memory 920 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it may perform the operations described in any one of the robust multimedia recommendation methods based on information bottlenecks in the above-mentioned embodiments.

[0162] The processor 910 reads and executes computer program instructions stored in the memory 920 to implement any one of the robust multimedia recommendation methods based on information bottleneck in the above embodiments.

[0163] In one example, the electronic device may further include a communication interface 930 and a bus 900. As shown in FIG4, the processor 910, the memory 920, and the communication interface 930 are connected via the bus 900 and communicate with each other.

[0164] The communication interface 930 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0165] Bus 900 includes hardware, software or both, and the parts of online data flow metering equipment are coupled to each other. For example, but not limitation, bus can include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 900 can include one or more buses. Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0166] In addition, in conjunction with the robust multimedia recommendation method based on information bottlenecks in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement any of the robust multimedia recommendation methods based on information bottlenecks in the above embodiments.

[0167] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0168] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0169] It should also be noted that the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0170] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0171] In summary, compared with the related art, this application has the following beneficial effects:

[0172] 1. This application uses information bottleneck constraints to eliminate the redundancy of the data itself, efficiently compress data information, forget other redundant information irrelevant to the recommendation task, learn the minimum multimedia content that meets the recommendation task, denoise the pre-trained multimedia features, and thus achieve more accurate multimedia recommendations.

[0173] 2. This application performs real-time analysis on the data in the user set and product set, and determines the user representation matrix and the product representation matrix to reconstruct the interaction matrix and calculate the reconstruction loss function; then, the information bottleneck loss function and the reconstruction loss function are combined in real time to perform multi-task learning, establish a multi-task optimization objective function, and achieve the accuracy and real-time performance of multimedia product recommendations based on the deep graph neural model.

[0174] 3. The multimedia denoising method based on information bottleneck proposed in this application is very lightweight, does not introduce any model parameters, and has strong practical value.

[0175] 4. This application uses the Hilbert-Schmidt independence constraint to construct an information bottleneck module, which is used to minimize the mutual information between multimedia content and its representation information. It can implicitly reduce the impact of multimedia noise on user modeling, improve the quality of user preference modeling, and improve the recommendation performance of multimedia products.

[0176] 5. The multimedia denoising method based on information bottleneck proposed in this application is universal and can be embedded in any recommendation method based on representation learning, and is applicable to many recommendation scenarios.

[0177] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A robust multimedia recommendation method based on information bottleneck, comprising: Construct heterogeneous data: user interaction matrix for the product, and product multimedia content data; Use pre-trained models to extract product multimedia content features; constructing a product association matrix according to the multimedia content features; Learning user and product representation matrices based on deep graph neural models; Based on the information bottleneck theory, the mutual information between multimedia content and its representation information is minimized to calculate the first loss function; reconstructing an interaction matrix based on the user representation matrix and the product representation matrix, and calculating a second loss function; Combine various loss functions for multi-task learning to update model parameters until the model converges.

2. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The method comprises: Step 1.1: Let U represent the user set, and U = {u1, ...u a ,...,u b ,...u M },u a represents the ath user, u b represents the b-th user, M represents the total number of users, 1≤a, b≤M; Let V denote the product set, and V = {v1, ..., v i ,...,v j ,...,v N }, v i represents the i-th product, v j represents the j-th product, N represents the total number of products, 1≤i, j≤N; Let r ai represents the a-th user u a For the i-th product v i The interaction data of users on products, then the user interaction matrix R = {r ai }M×N; Step 1.2, let C = {C 1 ,...,C k ,...,C K } represents the multimedia content of the product set, where C k represents the multimedia content of the kth modality in the product set V, and represents the multimedia content of the kth modality of the product set V, represents the i-th product v i The multimedia content of the kth modality, K represents the total number of modalities, 1≤k≤K; Use the pre-trained model of each modality to extract features of the corresponding modality multimedia content, and obtain the multimedia feature matrix set of K modalities {F 1 ,...,F k ,...,F K }, where F k represents the multimedia feature matrix of the kth modality; Step 3.1: Calculate the similarity matrix set {S 1 ,...,S k ,...,S K }: S k =L2(F k )×(L2(F k )) T (1) In formula (1), S k Represents the similarity matrix of the kth mode in the product set V, L2(F k ) represents the normalized multimedia feature matrix of the kth modality in the product set V, and T represents the matrix transpose; Step 3.2: According to formula (2), the similarity matrix set {S 1 ,...,S k ,...,S K } is sparsely processed to obtain the correlation matrix set {S′ 1 ,...,S′ k ,...,S′ K }; In formula (2), Represents the similarity matrix S of the kth mode in the product set V k The element value of the i-th row and j-th column of Represents the correlation matrix S′ of the kth mode in the product set V k The element value of the i-th row and j-th column of Represents the similarity matrix S of the kth mode in the product set V k All element values ​​in the i-th row, TopK′ represents the first K′ in descending order; Step 3.3: Calculate the correlation matrix S′ of the product set V according to formula (3): Step 3.4: Calculate the representation propagation matrix A according to formula (4): The normalized representation propagation matrix is ​​calculated by formula (5): In formula (5), D represents the degree matrix of matrix A; Step 4.1: Use Gaussian distribution to randomly initialize the user representation matrix P = {p1, ..., p a ,...,p M }, where p a represents the a-th user u a d1+Kd2-dimensional representation vector; where d1 represents the user collaborative representation dimension and d2 represents the content representation dimension; Use Gaussian distribution to randomly initialize the product co-representation matrix Q={q1,...,q i ,...,q N }, where q i represents the i-th product v i The d1-dimensional representation vector of Step 4.2: Calculate the multimedia content representation matrix T of each modality in the product set V by formula (6) = [T 1 ,...,T k ,...,T K ]: T k =MLP k (F k ) (6) In formula (6), MLP k is the kth multilayer perceptron, represents the multimedia content representation matrix of the kth modality; Step 4.3: Construct the product representation matrix E = [Q, T] and use the graph neural network to process {P, E} to obtain the user final representation matrix H u And the final representation matrix H v ; Step 5.1, use PCA algorithm to analyze the multimedia feature matrix set {F 1 ,...,F k ,...,F K } to reduce the dimension and obtain the multimodal feature matrix {F′ 1 ,...,F′ k ,...,F′ K },in, represents the kth mode F k The multimedia feature matrix after dimensionality reduction; Step 5.2: Construct the information bottleneck loss function L(θ1) according to formula (10): In formula (10), θ1 represents the parameters corresponding to K multi-layer perceptrons, and HSIC(,) represents the Hilbert-Schmidt independence index; Step 6.1: Predict the ath user u according to formula (11) a For the i-th product v i The interaction probability r′ ai : In formula (11), δ is the sigmoid activation function; Represents the user's final representation matrix H u The ath row in represents the transpose of the vector; Represents the final product representation matrix H v The i-th row in represents the vector; Step 6.2: Calculate the reconstruction loss function L(θ2) according to formula (12): In formula (12), θ2 represents all parameters involved in reconstructing the interaction matrix, and θ2 = [P, Q, θ1]; D a is the αth user u a All training data of D a ={(i, j)|i∈R a , j∈VR a }; R a represents the αth user u a A collection of interactive products; Represents the final product representation matrix H v The j-th row vector in ; Step 7.1: Establish the multi-task optimization objective L(θ) according to formula (13): L(θ)=L(θ1)+αL(θ2)+β||[P,Q,]|| 2 (13) In formula (13), θ = [P, Q, θ2] are all the parameters to be optimized, α and β are used to control the weights of the reconstruction loss function and the regularization term respectively; Step 7.2: Solve the optimization target L(θ) by gradient descent method to update the parameter θ until L(θ) converges to the minimum value, thereby obtaining the optimal parameter θ. * ; Step 7.3: Predict the ath user u according to formula (14) a For the i-th product v i The optimal interaction probability Thus we get the interaction matrix of user set U to product set V prediction Used to implement product recommendations: In formula (14), represents the a-th user u a The optimal representation vector of represents the i-th product v i The optimal representation vector of .

3. The robust multimedia recommendation method based on information bottleneck according to claim 2, wherein: The graph neural network in step 4.3 has a total of L convolutional layers, the current convolutional layer is l, and l=0 is initialized; Initialize the l-th layer node representation matrix H l ={P, E}; The l-th layer node representation matrix H l It is sent into the graph neural network, and the l+1 layer node representation matrix H is calculated according to formula (7) l+1 According to formula (8), the output of each graph convolution layer is aggregated to obtain the final node representation matrix H: According to formula (9), the user's final representation matrix H is obtained u And the final representation matrix H v : H u =H[:M],H v =H[M:] (9) In formula (9), H[:M] represents the 1st to Mth rows of the matrix H, and H[M:] represents the (M+1)th to (M+N)th rows of the matrix H.

4. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The structured heterogeneous data includes: Acquire a product set and a user set, wherein the product set includes the multimedia content data, and the user set includes a plurality of user data information corresponding to the multimedia content data; Determining a first mapping relationship between a plurality of users in the user set and each sub-data in the multimedia content data, and obtaining a plurality of first target data to represent an interaction relationship between the user set and the product set; Based on the multiple first target data, a user interaction matrix for the product is constructed.

5. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The pre-training model includes a plurality of preset sub-training models; The method of extracting multimedia content features of a product using a pre-trained model includes: Dividing the multimedia content data into multiple specific modalities, and establishing a second mapping relationship between the multiple specific modalities and the multiple sub-training models; Based on the second mapping relationship, the multimedia content features corresponding to each of the specific modalities in the product set are extracted through the multiple sub-training models to obtain multiple multimedia feature matrices and a multimedia feature matrix set.

6. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The multimedia content data corresponds to multiple specific modalities; The constructing of a product association matrix according to the multimedia content features includes: Analyzing similarities of the multiple specific modalities, and determining multiple similarity matrices and a similarity matrix set corresponding to the multiple specific modalities; Performing sparse processing and sorting on the plurality of similarity matrices to obtain a target sequence in which a plurality of elements are sequentially arranged; A predetermined number of larger elements are selected from the target sequence to determine a product association matrix and an association matrix set between a plurality of products.

7. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: After constructing the product association matrix according to the multimedia content features, the information bottleneck-based robust multimedia recommendation method further includes: Determine the interaction matrix of the user to the product as a first matrix subset, and transpose the first matrix subset, Get the second matrix subset; Determining the product association matrix as a third matrix subset; A first characterization propagation matrix is ​​constructed based on the first matrix subset, the second matrix subset, and the third matrix subset.

8. The robust multimedia recommendation method based on information bottleneck according to claim 7, wherein: After constructing the first representation propagation matrix based on the first matrix subset, the second matrix subset, and the third matrix subset, the information bottleneck-based robust multimedia recommendation method further includes: Calculating a degree matrix representing the first propagation matrix; A matrix multiplication operation is performed on the degree matrix and the first characterization propagation matrix to normalize the first characterization propagation matrix to obtain a second characterization propagation matrix.

9. The robust multimedia recommendation method based on information bottleneck according to claim 8, wherein: The user and product characterization matrix includes a user characterization matrix and a product characterization matrix; The user and product representation matrix is ​​learned based on the deep graph neural model, including: Randomly generate an initialized first user representation matrix and an initialized first product collaboration representation matrix through Gaussian distribution; Calculate the multimedia content representation matrix of the product through a preset multi-layer perceptron; fusing the first product collaboration representation matrix and the multimedia content representation matrix to determine a product representation matrix; Fusing the first user representation matrix and the product representation matrix to determine a layer 0 node representation matrix; Inputting the layer 0 node representation matrix into a preset graph neural network, using the second representation propagation matrix as an iteration coefficient, to obtain intermediate node representation matrices corresponding to multiple convolutional layers; Based on the number of convolutional layers in the graph neural network, multiple intermediate node representation matrices and the 0th layer node representation matrix are aggregated to obtain a final node representation matrix.

10. The robust multimedia recommendation method based on information bottleneck according to claim 9, wherein: The node final representation matrix H satisfies the expression: Where L is the number of convolutional layers, l represents the current convolutional layer, and H l Represents the node representation matrix of the lth layer.

11. The robust multimedia recommendation method based on information bottleneck according to claim 9, wherein: After aggregating the plurality of intermediate node representation matrices and the layer 0 node representation matrix based on the number of convolutional layers in the graph neural network to obtain a final node representation matrix, the information bottleneck-based robust multimedia recommendation method further includes: Analyze and extract each row vector in the final representation matrix of the node to determine a final representation matrix of the user and a final representation matrix of the product; The user's final representation matrix H u Satisfaction: H u =H[:M] Where H[:M] represents the 1st to Mth rows of the final node representation matrix; The final product characterization matrix H v Satisfaction: H v =H[M:] Where H[M:] represents the (M+1)th to (M+N)th rows of the final node representation matrix.

12. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The representation information of the multimedia content is a multimedia content representation matrix, and the first loss function is an information bottleneck loss function; and the step of minimizing the mutual information between the multimedia content and its representation information based on the information bottleneck theory and calculating the first loss function includes: Use PCA algorithm to reduce the dimension of multimedia feature matrix set to obtain the multimodal feature matrix after dimension reduction; Based on the information bottleneck theory, the Hilbert-Schmidt independence index between the multimodal feature matrix and the multimedia content representation matrix is ​​calculated to obtain an information bottleneck loss function.

13. The robust multimedia recommendation method based on information bottleneck according to claim 12, wherein: The information bottleneck loss function satisfies the expression: L(θ1) = ∑HSIC(F′ k , T k ) (16) Where L(θ1) represents the information bottleneck loss function, θ1 represents the parameters corresponding to K multi-layer perceptrons, HSIC(,) represents the Hilbert-Schmidt independence index, F′ k Represents the multimodal feature matrix after dimensionality reduction, T k represents the multimedia content representation matrix.

14. The robust multimedia recommendation method based on information bottleneck according to claim 11, wherein: The second loss function is a reconstruction loss function; reconstructing the interaction matrix based on the user representation matrix and the product representation matrix and calculating the second loss function includes: The user final representation matrix and the product final representation matrix are processed by a preset sigmoid activation function to respectively predict the interaction probability of each user to the product, so as to reconstruct the user-product interaction matrix; A reconstruction loss function is calculated based on the predicted multiple interaction probabilities and all parameters involved in reconstructing the interaction matrix.

15. The robust multimedia recommendation method based on information bottleneck according to claim 13, wherein: The heterogeneous data also includes a product set; the predicted interaction probability satisfies the expression: Where r′ ai is the interaction probability, δ is the sigmoid activation function, Represents the user's final representation matrix H u The ath row in represents the transpose of the vector; The final representation matrix H of the product is represented by v The i-th row in represents the vector; The reconstruction loss function L(θ2) satisfies the expression: Where θ2 represents all parameters involved in reconstructing the interaction matrix and corresponds to the reconstruction loss function. θ2 = [P, Q, θ1], where P is the first user representation matrix, Q is the first product collaboration representation matrix, and θ1 represents the parameters corresponding to K multi-layer perceptrons. D a is the ath user u a All training data of D a ={(i, j)|i∈R a , j∈VR a }; R a represents the a-th user u a The interactive product set, V represents the product set; The final representation matrix H of the product v The j-th row vector in .

16. The robust multimedia recommendation method based on information bottleneck according to claim 1, wherein: The first loss function is an information bottleneck loss function, and the second loss function is a reconstruction loss function; and the multi-task learning is performed by combining the loss functions to update the model parameters until the model converges, including: Based on the information bottleneck loss function and the reconstruction loss function, a multi-task optimization objective function is established; wherein the multi-task optimization objective function corresponds to all parameters to be optimized; Solving the multi-task optimization objective function by a gradient descent method to update all the parameters to be optimized; Determine the parameters corresponding to when the multi-task optimization objective function converges to the minimum value, and update all the parameters to be optimized. The new optimal parameters; Based on the optimal parameters, the target interaction probability of each user for the product is predicted respectively to determine a target interaction matrix, and the target interaction matrix is ​​used to implement product recommendation.

17. The robust multimedia recommendation method based on information bottleneck according to claim 16, wherein the multi-task optimization objective function satisfies the expression: L(θ)=L(θ1)+αL(θ2)+β||[P, Q,]|| 2 (19) Where θ = [P, Q, θ2] is all the parameters to be optimized, P is the first user representation matrix, Q is the first product collaborative representation matrix, and θ2 represents all the parameters corresponding to the reconstruction loss function. is a regularization term, α and β are weights for regulating the reconstruction loss function and the regularization term respectively; The target interaction matrix satisfies: in, Where, represents the a-th user u a The target representation vector, represents the i-th product v i The target representation vector.

18. An electronic device comprising: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the processor implements a robust multimedia recommendation method based on information bottlenecks, the method comprising: Construct heterogeneous data: user interaction matrix for the product, and product multimedia content data; Use pre-trained models to extract product multimedia content features; constructing a product association matrix according to the multimedia content features; Learning user and product representation matrices based on deep graph neural models; Based on the information bottleneck theory, the mutual information between multimedia content and its representation information is minimized to calculate the first loss function; reconstructing an interaction matrix based on the user representation matrix and the product representation matrix, and calculating a second loss function; Combine various loss functions for multi-task learning to update model parameters until the model converges.

19. A non-transitory readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the processor implements a robust multimedia recommendation method based on information bottleneck, the method comprising: Construct heterogeneous data: user interaction matrix for the product, and product multimedia content data; Use pre-trained models to extract product multimedia content features; constructing a product association matrix according to the multimedia content features; Learning user and product representation matrices based on deep graph neural models; Based on the information bottleneck theory, the mutual information between multimedia content and its representation information is minimized to calculate the first loss function; reconstructing an interaction matrix based on the user representation matrix and the product representation matrix, and calculating a second loss function; Combine various loss functions for multi-task learning to update model parameters until the model converges.

Images