A multi-modal product recommendation method based on cross-modal correction

Abstract

The application relates to a multi-modal product recommendation method based on cross-modal correction, and belongs to the technical field of product recommendation. A product data set and a user data set are acquired, a product relation graph and a product user relation graph are constructed, picture information and text information of products are corrected, target picture feature vectors, target text feature vectors and modal feature vectors are output, graph convolution is performed on the product relation graph and the product user relation graph, target modal feature vectors, product feature representations and user feature representations are obtained, user behavior feature vectors and shared modal feature vectors are extracted, based on the behavior feature vectors, the shared modal feature vectors and the target modal feature vectors, modal preference feature representations of users for products are obtained, product target feature representations are obtained based on the product feature representations and the modal preference feature representations, user target feature representations are obtained based on the user feature representations and the modal preference feature representations, preference scores of the users for the products are calculated, and product recommendation is performed on the users.

Description

Technical Field

[0001] This invention relates to the field of product recommendation technology, and in particular to a multimodal product recommendation method based on cross-modal correction. Background Technology

[0002] Recommender systems are becoming indispensable tools in many fields. They primarily make recommendations by mining the binary relationship between users and products. Multimodal recommendation, leveraging the advantages of multimodal information (such as images and text), is emerging as a new trend in recommender systems. Research shows that models utilizing multimodal product information have better recommendation capabilities than those using a single modality.

[0003] Early multimodal recommender systems were often based on deep learning. For example, Bayesian Ranking (VBPR) extracts visual features of products through pre-trained convolutional neural networks and incorporates these features into the product feature representation. It also utilizes users' historical behavior data to understand user preferences for recommendations. Recently, with the rise of graph convolutional networks (GCNNs) in recommender systems, most research has shifted to GCNNs. MMGCN models and learns representations of multimodal product information to obtain multimodal product representations, which are then combined with users' historical behavior data for product recommendations. DualGNN utilizes the user-product relationship graph to extract user and product feature representations, considering the interaction between users and products to provide more accurate product recommendations. However, most existing methods ignore the impact of users' preferences for different product modalities on product recommendation results. For example, user A may buy product B because they are attracted by the image information of product B, while user A may buy product C because they are attracted by the text information of product C. Furthermore, when extracting product features, existing methods process different modal information of products separately, ignoring the correlation between different modal information, which makes the extracted product feature information inaccurate, thereby reducing the accuracy of product recommendation results.

[0004] In summary, existing multimodal product recommendation methods ignore the correlation between different modal information when extracting product features, resulting in inaccurate product features. Furthermore, they do not consider users' preferences for different modal information when making product recommendations, thus failing to provide users with more accurate and personalized product recommendations. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology ignores the correlation between different modal information when extracting product features, resulting in inaccurate product features, and fails to consider users' preferences for different modal information when making product recommendations, thus failing to provide users with more accurate and personalized product recommendations.

[0006] To address the aforementioned technical problems, this invention provides a multimodal product recommendation method based on cross-modal correction, comprising:

[0007] Obtain a product dataset containing text information, image information, and product ID vectors of multiple products, and construct a product relationship graph based on the product dataset; obtain a user dataset containing historical interaction data and user ID vectors of multiple users, and construct a product-user relationship graph based on the product dataset and the user dataset;

[0008] The product ID vector, image information, and text information of each product in the product dataset are input into the cross-modal correction module of the trained multimodal product recommendation model, and the target image feature vector and target text feature vector of each product are output. Based on the target image feature vector and target text feature vector of each product, the modal feature vector of each product is obtained.

[0009] The modal feature vectors of each product and the product relationship graph are input into the view encoder module of the trained multimodal product recommendation model, and the target modal feature vectors of each product are output; the product user relationship graph is input into the view encoder module, and the product feature representations of each product and the user feature representations of each user are output;

[0010] The adaptive preference module in the trained multimodal product recommendation model is used to extract the user behavior feature vector of each user's user ID vector and the shared modal feature vector of each product's target modal feature vector. The user behavior feature vector, the shared modal feature vector of each product, and the target modal feature vector of each product are input into the adaptive preference module, and the modal preference feature representation of each user for each product is output.

[0011] Based on the product feature representation and modal preference feature representation of each product, the product target feature representation of each product is obtained. Based on the user feature representation and modal preference feature representation of each user, the user target feature representation of each user is obtained. Based on the user target feature representation and the product target feature representation, the preference score between the user and the product is calculated so as to recommend products to the user based on the preference score.

[0012] In one embodiment of the present invention, the product ID vector, image information, and text information of each product in the product dataset are input into the cross-modal correction module of a trained multimodal product recommendation model, and the target image feature vector and target text feature vector of each product are output. The modal feature vector of each product is obtained based on the target feature vector and target text feature vector of each product, including:

[0013] The MLP neural network in the cross-modal correction module is used to extract features from the image and text information of each product, and outputs high-order image feature vectors and high-order text feature vectors for each product.

[0014] The high-order text feature vector of each product is corrected using the product ID vector of each product to obtain the target text feature vector of each product. The high-order image feature vector of each product is then corrected using the target text feature vector of each product to obtain the target image feature vector of each product.

[0015] The modal feature vectors of each product are obtained by combining the target text feature vector and the target image feature vector.

[0016] In one embodiment of the present invention, the high-order image feature vectors of each product are:

[0017] e i ′ ,v =W1e i,v +b1,

[0018] Among them, e i ′ ,v Let W1 represent the high-order image feature vector of the i-th product, b1 be the first weight matrix, and e be the first bias matrix. i,v This represents the image information of the i-th product;

[0019] The high-order text feature vectors for each product are as follows:

[0020] e i ′ ,t =W1e i,t +b1,

[0021] Among them, e i ′ ,t Let e ​​represent the high-order text feature vector of the i-th product. i,t This represents the text information of the i-th product;

[0022] The target text feature vectors for each product are as follows:

[0023] ei ′,′ t =Gate t (e i,id ,e i ′ ,t ) = e i,id ⊙σ(W2e i ′ ,t +b2),

[0024] Among them, e i ′,′ t Let e ​​represent the target text feature vector of the i-th product. i,id Let represent the product ID vector of the i-th product, ⊙ represent the element-wise product, σ is the sigmoid non-linear activation function, W2 is the second weight matrix, and b2 is the second bias matrix;

[0025] The feature vectors of the target images for each product are as follows:

[0026] e i ′,′ v =Gate v (e i ′,′ t ,e i ′ ,v ) = e i ′,′ t ⊙σ(W3e i ′ ,v +b3),

[0027] Among them, e i ′,′ v Let represent the feature vector of the target image of the i-th product, ⊙ represent the element-wise product, σ is the sigmoid non-linear activation function, W3 is the third weight matrix, and b3 is the third bias matrix.

[0028] In one embodiment of the present invention, the modal feature vectors of each product and the product relationship graph are input into the view encoder module of the trained multimodal product recommendation model, and the target modal feature vectors of each product are output, including:

[0029] Calculate the similarity between each product and its neighboring products in the product relationship graph, and perform KNN sparsification on the product relationship graph. Based on the similarity between each product and its neighboring products after sparsification, obtain the similarity matrix of the product relationship graph.

[0030] Based on the similarity matrix and the modal feature vector of each product, the product modal feature vector of each product is obtained, and the modal preference feature vector of the user is calculated based on the product modal feature vector of each product.

[0031] The target modal feature vector of each product is obtained by concatenating the product modal feature vector and the user modal preference feature vector.

[0032] In one embodiment of the present invention, the formula for calculating the similarity between each product and its adjacent products in the product relationship diagram is as follows:

[0033]

[0034] in, This represents the similarity between the i-th product and its adjacent products in the product relationship graph. ′ In the product relationship diagram, e represents the product adjacent to the i-th product. i ′,′ m Let e ​​represent the modal feature vector of the i-th product. i " ′ ,m Indicates the i-th ′ The modal feature vectors of each product, where T represents the transpose of the matrix;

[0035] The formula for calculating the product modal feature vector of each product is as follows:

[0036]

[0037] in, S represents the product modal feature vector of the i-th product. ′ m This represents the similarity matrix of the product relationship diagram under mode m;

[0038] The formula for calculating the user's modal preference feature vector is:

[0039]

[0040] in, This represents the modal preference feature vector of the u-th user. This represents the set of products that user u has interacted with. This represents the set of users who have interacted with product i.

[0041] In one embodiment of the present invention, inputting the product user relationship diagram into the view encoder module and outputting the product feature representation of each product and the user feature representation of each user includes:

[0042] The probability of each connection line in the product user relationship graph being pruned is calculated based on the degree-sensitive probability pruning strategy. The product user relationship graph is pruned based on the probability of each connection line being pruned and the preset pruning ratio. A symmetric adjacency matrix is ​​constructed based on the pruned product user relationship graph.

[0043] The view encoder module is used to perform graph convolution on the symmetric adjacency matrix to output the product feature representation of each product and the user feature representation of each user.

[0044] The formula for calculating the probability of each connecting line in the product user relationship diagram being pruned is as follows:

[0045]

[0046] Where, p k w represents the probability that the k-th connection in the product-user relationship graph is pruned. a w represents the degree of node a in the product-user relationship graph. b This represents the degree of node b in the product user relationship graph, where node a and node b are the two nodes of the k-th connecting line;

[0047] The product characteristics of each product are represented as follows:

[0048]

[0049] Among them, e i,id Let L represent the product feature representation of the i-th product, and let L represent the number of layers in the graph convolution operation. This represents the output of the convolution operation on the l-th graph;

[0050]

[0051] in, This represents the set of products that user u has interacted with. This represents the set of users who have interacted with product i. This represents the user feature representation of the u-th user, as output by the graph convolution operation in the (l-1)-th layer.

[0052] The user characteristics of each user are represented as follows:

[0053]

[0054] Among them, e u,id This represents the user characteristics of the u-th user. This represents the output of the convolution operation on the l-th graph;

[0055]

[0056] in, This represents the product feature representation of the i-th product output by the (l-1)-th layer graph convolution operation.

[0057] In one embodiment of the present invention, the adaptive preference module in the trained multimodal product recommendation model extracts the user behavior feature vector of each user's user ID vector and the shared modal feature vector of each product's target modal feature vector, respectively; and inputs the user behavior feature vector, the shared modal feature vector of each product, and the target modal feature vector of each product into the adaptive preference module, outputting the modal preference feature representation of each user for each product, including:

[0058] The feature extraction submodule in the adaptive preference module is used to extract the behavioral feature vector of each user's user ID vector;

[0059] The attention submodule in the adaptive preference module is used to extract the shared modal feature vector of the target modal feature vector of each product;

[0060] The behavioral feature vector of each user, the shared modal feature vector of each product, and the target modal feature vector of each product are input into the feature fusion submodule in the adaptive preference module, and the modal preference feature representation of each user for each product is output.

[0061] In one embodiment of the present invention, the behavioral feature vector of each user is:

[0062] P u,m =σ(W4e u,id +b4),

[0063] Among them, P u,m Let σ represent the behavioral feature vector of the u-th user, σ be the sigmoid non-linear activation function, W4 be the fourth weight matrix, and e u,id b4 represents the user ID vector of the u-th user, and b4 is the fourth bias matrix;

[0064] The shared modal feature vectors of all products are:

[0065]

[0066] Among them, E i,s Let M represent the shared modal feature vector of the i-th product, and let M represent the modality set. i,m This represents the attention score of the target modal feature vector of the i-th product. This represents the target modal feature vector of the i-th product;

[0067]

[0068] Where W5 is the fifth weight matrix, b5 is the fifth bias matrix, q1 is the attention vector, and softmax and tanh are non-linear activation functions;

[0069] Each user's modal preference characteristics for each product are represented as follows:

[0070]

[0071] Among them, E f Let represent the modal preference feature representation of the u-th user for the i-th product, and ⊙ represent the element-wise product.

[0072] In one embodiment of the present invention, the target features of each product are represented as follows:

[0073] e i =e i,f +e i,id ,

[0074] Among them, e i Let e ​​represent the target feature of the i-th product. i,id Let e ​​represent the product feature representation of the i-th product. i,f This represents the product representation of the i-th product in the modal preference feature representation;

[0075] The user's target characteristics are represented as follows:

[0076] e u =e u,f +e u,id ,

[0077] Among them, e u Let e ​​represent the user target feature of the u-th user. u,id Let e ​​represent the user characteristics of the u-th user. u,f This represents the user representation of the u-th user in the modal preference feature representation;

[0078] The formula for calculating the user's preference score for the product is:

[0079]

[0080] in, Let represent the preference score of the u-th user for the i-th product, and T represent the transpose of the matrix.

[0081] In one embodiment of the present invention, the training process of the multimodal product recommendation model includes:

[0082] Multiple triples containing positive and negative samples were constructed based on the product and user datasets in the training set. Multiple triples are input into the multimodal product recommendation model for iterative training until the product recommendation loss function is minimized, thus obtaining the trained multimodal product recommendation model.

[0083] Among them, the triplet u represents the u-th user in the user dataset, i represents the i-th product that user u clicked in the product dataset, and j represents the j-th product that user u did not click in the product dataset;

[0084] The product recommendation loss function is:

[0085]

[0086] in, Let Θ represent the product recommendation loss function, and ||Θ||² represent L2 regularization. Represents the BPR loss function. Let λ represent the loss function for the self-supervised task. c Let λ be the first hyperparameter. reg This is the second hyperparameter;

[0087] BPR loss function for:

[0088]

[0089] in, This represents the preference score of the u-th user for the i-th product. This represents the preference score of the u-th user for the j-th product;

[0090] Self-supervised task loss function for:

[0091]

[0092] Where U represents the user dataset, I represents the product dataset, and e u,id Let e ​​represent the first feature representation of the u-th user in the user dataset. u,f Let τ represent the second feature representation of the u-th user in the user dataset, and let e represent the temperature hyperparameter of the softmax function. i,id Let e ​​represent the first feature representation of the i-th product in the product dataset. i,f This represents the second feature representation of the i-th product in the product dataset.

[0093] The multimodal product recommendation method based on cross-modal correction provided by this invention utilizes a cross-modal correction module to correct the text and image information of products, mining the correlation between different modal information to make the extracted modal features as consistent as possible, thereby improving the accuracy of product feature extraction. A view encoder performs graph convolution on the product relationship graph, and further extracts product modal features based on the output of the view encoder and the corrected product modal features. Simultaneously, a view encoder performs graph convolution on the product-user relationship graph to extract product feature representations and user feature representations containing user-product relationships. Then, an adaptive preference module extracts user behavioral features and shared modal features of products respectively. Based on user behavioral features, shared modal features, and modal features, a modal preference feature representation containing user preferences for products is obtained. Based on the product feature representation and modal preference feature representation, a product target feature representation is obtained. Based on the user feature representation and modal preference feature representation, a user target feature representation is obtained. Finally, a preference score between users and products is calculated using the user target feature representation and product target feature representation to recommend products to users. This application not only considers the correlation between different modal information of the product, improving the accuracy of product feature extraction, but also fully considers the user's modal information preference by extracting the modal feature preference features of the product, thus improving the accuracy of product recommendation results. Attached Figure Description

[0094] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0095] Figure 1 A flowchart of the multimodal product recommendation method based on cross-modal correction provided in this application;

[0096] Figure 2 A schematic diagram illustrating the test results of the KNN sparsity parameter K provided in this application on different datasets; wherein, Figure 2 (a) in the figure is a schematic diagram of the test results of parameter K on the Baby dataset. Figure 2 (b) in the figure is a schematic diagram of the test results of parameter K on the Sports dataset. Figure 2 (c) in the figure is a schematic diagram of the test results of parameter K on the Clothing dataset;

[0097] Figure 3 A schematic diagram illustrating the test results of the preset degree-sensitive pruning ratio ρ of the degree-sensitive pruning strategy provided in this application on different datasets; wherein, Figure 3 (a) in the figure is a schematic diagram of the test results of the preset degree-sensitive pruning ratio ρ on the Baby dataset. Figure 3(b) in the figure is a schematic diagram of the test results of the preset degree-sensitive pruning ratio ρ on the Sports dataset. Figure 3 (c) in the figure is a schematic diagram of the test results of the preset degree-sensitive pruning ratio ρ on the Clothing dataset;

[0098] Figure 4 The first hyperparameter λ provided for this application c Schematic diagram of test results on different datasets; where, Figure 4 In this context, (a) represents the first hyperparameter λ. c A diagram illustrating the test results on the Baby dataset. Figure 4 In the first hyperparameter λ, (b) represents the first hyperparameter λ. c A diagram illustrating the test results on the Sports dataset. Figure 4 (c) in the equation represents the first hyperparameter λ. c A diagram illustrating test results on the Clothing dataset;

[0099] Figure 5 This is a schematic diagram of the multimodal product recommendation model structure provided in this application;

[0100] Figure 6 This application provides a schematic diagram of the results of ablation experiments on a CMC-MMR model; wherein, Figure 6 (a) in the figure is a schematic diagram of the recall rate of the ablation experiment on various datasets. Figure 6 (b) in the figure is a schematic diagram of the normalized cumulative loss gain index of the ablation experiment on various datasets;

[0101] Figure 7 This is a schematic diagram of the modal feature distribution of the image before correction provided in this application; wherein, Figure 7 (a) in the image is a scatter plot of the modal feature distribution. Figure 7 (b) in the diagram is a schematic diagram of the modal feature distribution density of the image. Figure 7 (c) in the figure is a schematic diagram of the Gaussian kernel density estimation results of the image modal features;

[0102] Figure 8 This is a schematic diagram of the modal feature distribution of the corrected image provided in this application; wherein, Figure 8 (a) in the image is a scatter plot of the modal feature distribution. Figure 8 (b) in the diagram is a schematic diagram of the modal feature distribution density of the image. Figure 8 (c) in the figure is a schematic diagram of the Gaussian kernel density estimation results of the image modal features;

[0103] Figure 9 This application provides a schematic diagram of the distribution of text modal features before correction; wherein, Figure 9 (a) in the figure is a scatter plot of the text modal feature distribution. Figure 9 (b) in the diagram is a schematic diagram of the distribution density of text modal features. Figure 9 (c) in the figure is a schematic diagram of the Gaussian kernel density estimation results of text modal features;

[0104] Figure 10 This is a schematic diagram of the corrected text modal feature distribution provided in this application; wherein, Figure 10 (a) in the figure is a scatter plot of the text modal feature distribution. Figure 10 (b) in the diagram is a schematic diagram of the distribution density of text modal features. Figure 10 (c) in the figure is a schematic diagram of the Gaussian kernel density estimation results of the text modal features. Detailed Implementation

[0105] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0106] like Figure 1 As shown, the multimodal product recommendation method based on cross-modal correction provided in this application includes:

[0107] S10: Obtain a product dataset containing text information, image information, and product ID vectors of multiple products, and construct a product relationship graph based on the product dataset; Obtain a user dataset containing historical interaction data and user ID vectors of multiple users, and construct a product-user relationship graph based on the product dataset and the user dataset;

[0108] S20: Input the product ID vector, image information and text information of each product in the product dataset into the cross-modal correction module of the trained multimodal product recommendation model, output the target image feature vector and target text feature vector of each product, and obtain the modal feature vector of each product based on the target image feature vector and target text feature vector of each product;

[0109] Optionally, in other embodiments of this application, the modal features of the product are not limited to the image modal features and text modal features of the product, but can also include other modal features of the product, such as the tactile modal features, video modal features, etc.

[0110] S30: Input the modal feature vectors and product relationship graphs of each product into the view encoder module of the trained multimodal product recommendation model, and output the target modal feature vectors of each product; input the product user relationship graphs into the view encoder module, and output the product feature representations of each product and the user feature representations of each user;

[0111] S40: Utilize the adaptive preference module in the trained multimodal product recommendation model to extract the user behavior feature vector of each user's user ID vector and the shared modal feature vector of each product's target modal feature vector; input each user's behavior feature vector, each product's shared modal feature vector, and each product's target modal feature vector into the adaptive preference module, and output each user's modal preference feature representation for each product;

[0112] S50: Based on the product feature representation and modal preference feature representation of each product, obtain the product target feature representation of each product; based on the user feature representation and modal preference feature representation of each user, obtain the user target feature representation of each user; calculate the preference score between the user and the product based on the user target feature representation and the product target feature representation, so as to recommend products to users based on the preference score.

[0113] This application utilizes a cross-modal correction module to correct the text and image information of products, mining the correlation between different modal information to make the extracted modal features as consistent as possible, thereby improving the accuracy of product feature extraction. A view encoder performs graph convolution on the product relationship graph, and based on the output of the view encoder and the corrected product modal features, further product modal features are extracted. Simultaneously, a view encoder performs graph convolution on the product-user relationship graph, extracting product feature representations and user feature representations containing user-product relationships. Then, an adaptive preference module extracts user behavioral features and shared modal features of products respectively. Based on user behavioral features, shared modal features, and modal features, a modal preference feature representation containing user preferences for products is obtained. Based on the product feature representation and modal preference feature representation, a product target feature representation is obtained. Based on the user feature representation and modal preference feature representation, a user target feature representation is obtained. Finally, a preference score between users and products is calculated using the user target feature representation and product target feature representation to provide product recommendations to users. This application not only considers the correlation between different modal information of the product, improving the accuracy of product feature extraction, but also fully considers the user's modal information preference by extracting the modal feature preference features of the product, thus improving the accuracy of product recommendation results.

[0114] Specifically, in some embodiments of this application, the product ID vector and user ID vector in step S10 are unique identifiers of the product or the user, respectively; the user's historical interaction data represents the products that the user has clicked and the products that the user has not clicked.

[0115] Since multimodal product recommendations are contaminated by original noise when using raw modal data, and processing information for each modality separately ignores the relationships between modalities, this application embodiment employs a cross-modal correction module to correct the product's image modality through text modality, so that the product's image modal features conform as closely as possible to the information described in the text modality, thereby achieving denoising of the original modal features. Specifically, step S20 includes:

[0116] S201: Use the MLP neural network in the cross-modal correction module to extract features from the image and text information of each product, and output the high-order image feature vector and high-order text feature vector of each product;

[0117] The high-order image feature vectors for each product are as follows:

[0118] e i ′ ,v =W1e i,v +b1,

[0119] Among them, e i ′ ,v Let W1 represent the high-order image feature vector of the i-th product, b1 be the first weight matrix, and e be the first bias matrix. i,v This represents the image information of the i-th product;

[0120] The high-order text feature vectors for each product are as follows:

[0121] e i ′ ,t =W1e i,t +b1,

[0122] Among them, e i ′ ,t Let e ​​represent the high-order text feature vector of the i-th product. i,t This represents the text information of the i-th product;

[0123] S202: Use the product ID vector of each product to correct the high-order text feature vector of each product to obtain the target text feature vector of each product. Use the target text feature vector of each product to correct the high-order image feature vector of each product to obtain the target image feature vector of each product.

[0124] The target text feature vectors for each product are as follows:

[0125] e i ′,′ t =Gate t(e i,id ,e i ′ ,t ) = e i,id ⊙σ(W2e i ′ ,t +b2),

[0126] Among them, e i ′,′ t Let e ​​represent the target text feature vector of the i-th product. i,id Let represent the product ID vector of the i-th product, ⊙ represent the element-wise product, σ is the sigmoid non-linear activation function, W2 is the second weight matrix, and b2 is the second bias matrix;

[0127] The feature vectors of the target images for each product are as follows:

[0128] e i ′,′ v =Gate v (e i ′,′ t ,e i ′ ,v ) = e i ′,′ t ⊙σ(W3e i ′ ,v +b3),

[0129] Among them, e i ′,′ v Let represent the feature vector of the target image of the i-th product, ⊙ represent the element-wise product, σ is the sigmoid non-linear activation function, W3 is the third weight matrix, and b3 is the third bias matrix.

[0130] S203: Combine the target text feature vector and target image feature vector of each product to obtain the modal feature vector of each product.

[0131] Alternatively, when other modal features are involved, text modal features can also be used to correct other modal features.

[0132] Furthermore, performing graph convolution on the product relationship graph can capture semantically relevant signals, thereby enriching the modal features of the products. Therefore, in this embodiment, the view encoder module is used to convolve the product relationship graph to further extract the modal features of the products. However, if each product establishes relationships with other products, the resources consumed will be very large, and establishing connections between unrelated products will introduce unnecessary noise. Therefore, in step S30 of this application, the modal feature vectors of each product and the product relationship graph are input into the view encoder module in the trained multimodal product recommendation model, and the target modal feature vectors of each product are output, including:

[0133] S301: Calculate the similarity between each product and its neighboring products in the product relationship graph, and perform KNN sparsification on the product relationship graph. Based on the similarity between each product and its neighboring products after sparsification, obtain the similarity matrix of the product relationship graph.

[0134] The formula for calculating the similarity between each product and its adjacent products is as follows:

[0135]

[0136] in, This represents the similarity between the i-th product and its adjacent products in the product relationship graph. ′ In the product relationship diagram, e represents the product adjacent to the i-th product. i ′,′ m Let e ​​represent the modal feature vector of the i-th product. i " ′ ,m Indicates the i-th ′ The modal feature vectors of each product, where T represents the transpose of the matrix;

[0137] S302: Based on the similarity matrix and the modal feature vector of each product, obtain the product modal feature vector of each product, and calculate the user's modal preference feature vector based on the product modal feature vector of each product;

[0138] The formula for calculating the product modal feature vector of each product is as follows:

[0139]

[0140] in, S represents the product modal feature vector of the i-th product. ′ m This represents the similarity matrix of the product relationship diagram under mode m;

[0141] The formula for calculating the user's modal preference feature vector is:

[0142]

[0143] in, This represents the modal preference feature vector of the u-th user. This represents the set of products that user u has interacted with. This represents the set of users who have interacted with product i.

[0144] S303: Concatenate the product modal feature vector of each product with the modal preference feature vector of the user to obtain the target modal feature vector of each product.

[0145] For example, KNN sparsification retains the K connections with the highest similarity to each product in the product relationship graph:

[0146]

[0147] This application embodiment also tested parameter K on three datasets, and the test results are as follows: Figure 2 As shown, for all datasets, the best performance is achieved when K=10. However, for other K values, recommendation performance declines to some extent. In particular, for the Sports and Clothing datasets, a sharp drop occurs when K=15 compared to K=10. This is because increasing K introduces too many irrelevant products into the Sports and Clothing datasets, which have a high number of interactions, thus leading to a decrease in recommendation performance. K may be affected differently in different scenarios, and a smaller K can prevent modal noise contamination.

[0148] Optionally, in some embodiments of this application, in order to alleviate the gradient explosion problem, the sparsified product relationship graph can also be normalized.

[0149] Furthermore, in order to capture the relationship between users and products and obtain higher-order feature signals, this application also uses a view encoder module to perform graph convolution on the product-user relationship graph. Specifically, in step S30, the product-user relationship graph is input to the view encoder module, and the output of product feature representations for each product and user feature representations for each user includes:

[0150] S304: Calculate the probability of each connection line in the product user relationship graph being pruned based on the degree-sensitive probability pruning strategy, prune the product user relationship graph based on the probability of each connection line being pruned and the preset pruning ratio, and construct a symmetric adjacency matrix based on the pruned product user relationship graph.

[0151] S305: Use the view encoder module to perform graph convolution on the symmetric adjacency matrix to output the product feature representation of each product and the user feature representation of each user.

[0152] Specifically, in this embodiment, LightGCN is used for graph convolution, which can eliminate feature transformation and nonlinear activation modules to simplify graph convolution operations. After L layers of message passing, the representation of each GCN layer is combined by element-wise summation. Furthermore, since the original user product representation has not undergone graph convolution and contains noise, this application only aggregates the product feature representation and user feature representation from the 1st to the Lth layers.

[0153] The formula for calculating the probability of each connecting line in the product user relationship diagram being pruned is as follows:

[0154]

[0155] Where, p k w represents the probability that the k-th connection in the product-user relationship graph is pruned. a w represents the degree of node a in the product-user relationship graph. b This represents the degree of node b in the product user relationship graph, where node a and node b are the two nodes of the k-th connecting line;

[0156] The product characteristics of each product are represented as follows:

[0157]

[0158] Among them, e i,id Let L represent the product feature representation of the i-th product, and let L represent the number of layers in the graph convolution operation. This represents the output of the convolution operation on the l-th graph;

[0159]

[0160] in, This represents the user feature representation of the u-th user, as output by the graph convolution operation in the (l-1)-th layer.

[0161] The user characteristics of each user are represented as follows:

[0162]

[0163] Among them, e u,id This represents the user characteristics of the u-th user. This represents the output of the convolution operation on the l-th graph;

[0164]

[0165] in, This represents the product feature representation of the i-th product output by the (l-1)-th layer graph convolution operation.

[0166] Specifically, since popular nodes in graph convolutional neural networks may suffer from over-smoothing, this embodiment first uses a degree-sensitive pruning strategy to prune the product-user relationship graph. Specifically, the pruned product-user relationship graph R... ρ It can be represented as:

[0167] R ρ =Dropout(R),

[0168] Where Dropout represents a degree-sensitive pruning strategy, and R represents a product user relationship graph;

[0169] The symmetric adjacency matrix A obtained based on the pruned product-user relationship graph is:

[0170]

[0171] Where 0 represents a matrix with all zeros;

[0172] This application embodiment also tested the preset sensitivity pruning ratio ρ on three datasets, and the test results are as follows: Figure 3 As shown in the figure, the optimal pruning ratio for the Baby dataset is 0.7, while the optimal pruning ratio for the Sports and Clothing datasets is 0.8. For the Baby dataset, the total number of user product interactions is the smallest. An excessively high pruning ratio will cause a sharp decrease in the number of user product interactions, resulting in the loss of some necessary information, which will affect the recommendation performance of the recommendation system. Conversely, the Sports and Clothing datasets have more user product interactions. If a smaller pruning ratio is used, the pruned graph will still contain a certain number of interactions, retaining more irrelevant edges. During graph encoding, these irrelevant edges will affect the recommendation performance of the model.

[0173] Furthermore, in order to understand users' preferences for product modalities and improve the model's recommendation performance, this application uses an adaptive preference module to extract users' behavioral features and assigns different weights to each product modal feature. Specifically, step S40 includes the following steps:

[0174] S401: Extract the behavioral feature vector of each user's user ID vector using the feature extraction submodule in the adaptive preference module;

[0175] The behavioral feature vectors of each user are as follows:

[0176] P u,m =σ(W4e u,id +b4),

[0177] Among them, P u,mLet σ represent the behavioral feature vector of the u-th user, σ be the sigmoid non-linear activation function, W4 be the fourth weight matrix, and e u,id b4 represents the user ID vector of the u-th user, and b4 is the fourth bias matrix;

[0178] S402: Extract the shared modal feature vector of the target modal feature vector for each product using the attention submodule in the adaptive preference module;

[0179] The shared modal feature vectors of all products are:

[0180]

[0181] Among them, E i,s Let M represent the shared modal feature vector of the i-th product, and let M represent the modality set. i,m This represents the attention score of the target modal feature vector of the i-th product. This represents the target modal feature vector of the i-th product;

[0182] in, W5 is the fifth weight matrix, b5 is the fifth bias matrix, q1 is the attention vector, and softmax and tanh are non-linear activation functions;

[0183] S403: Input each user's behavioral feature vector, each product's shared modal feature vector, and each product's target modal feature vector into the feature fusion submodule in the adaptive preference module, and output each user's modal preference feature representation for each product;

[0184] Each user's modal preference characteristics for each product are represented as follows:

[0185]

[0186] Among them, E f Let represent the modal preference feature representation of the u-th user for the i-th product, and ⊙ represent the element-wise product.

[0187] Based on the above embodiments, the product target features of each product in step S50 are represented as follows:

[0188] e i =e i,f +e i,id ,

[0189] Among them, e i Let e ​​represent the target feature of the i-th product. i,id Let e ​​represent the product feature representation of the i-th product. i,f This represents the product representation of the i-th product in the modal preference feature representation;

[0190] The user's target characteristics are represented as follows:

[0191] e u =e u,f +e u,id ,

[0192] Among them, e u Let e ​​represent the user target feature of the u-th user. u,id Let e ​​represent the user characteristics of the u-th user. u,f This represents the user representation of the u-th user in the modal preference feature representation;

[0193] The formula for calculating the user's preference score for the product is:

[0194]

[0195] in, Let represent the preference score of the u-th user for the i-th product, and T represent the transpose of the matrix.

[0196] This application embodiment also includes training a multimodal product recommendation model, the specific process of which includes:

[0197] Multiple triples containing positive and negative samples were constructed based on the product and user datasets in the training set. Multiple triples are input into the multimodal product recommendation model for iterative training until the product recommendation loss function is minimized, thus obtaining the trained multimodal product recommendation model.

[0198] Among them, triplet u represents the u-th user in the user dataset, i represents the i-th product that user u clicked in the product dataset, and j represents the j-th product that user u did not click in the product dataset;

[0199] The product recommendation loss function is:

[0200]

[0201] in, Let Θ represent the product recommendation loss function, and ||Θ||² represent L2 regularization. Represents the BPR loss function. Let λ represent the loss function for the self-supervised task. c Let λ be the first hyperparameter. reg This is the second hyperparameter;

[0202] BPR loss function for:

[0203]

[0204] in, This represents the preference score of the u-th user for the i-th product. This represents the preference score of the u-th user for the j-th product;

[0205] Self-supervised task loss function for:

[0206]

[0207] Where U represents the user dataset, I represents the product dataset, and e u,id Let e ​​represent the first feature representation of the u-th user in the user dataset. u,f Let τ represent the second feature representation of the u-th user in the user dataset, and let e represent the temperature hyperparameter of the softmax function. i,id Let e ​​represent the first feature representation of the i-th product in the product dataset. i,f This represents the second feature representation of the i-th product in the product dataset;

[0208] Specifically, for the first hyperparameter λ that controls the importance of the self-supervised task... c In this application embodiment, λ was tested on three datasets. c The impact on the model was observed to assess the influence of the importance of the self-supervised task on the model's recommendation performance. The test results are as follows: Figure 4 As shown, set a reasonable parameter λ c For the Baby dataset, the optimal λ can effectively improve recommendation performance. c The value is approximately 0.01 for the Sports and Clothing datasets, and approximately 0.1 for the Clothing dataset. This suggests that as the dataset becomes larger, the number of users also increases, necessitating an increase in the importance of the self-supervised preference module to improve the recommendation performance of the recommendation system. However, for the smaller Baby dataset, the importance of the self-supervised module is relatively weak.

[0209] For example, such as Figure 5 The diagram shown is a structural schematic of the trained multimodal product recommendation model provided in the embodiment of this application.

[0210] To verify the effectiveness of the method provided in this application, this embodiment also conducted experiments on three datasets using this model. Table 1 shows the dataset information:

[0211] Table 1

[0212] Baby 19445 7050 160792 0.117％ Sports 35598 18357 296337 0.045％ Clothing 39387 23033 278677 0.031％

[0213] The dataset selected in this embodiment is Baby, Sports and Outdoors, Clothing Shoes and Jewelry. The dataset contains textual and image information of the products. In this embodiment, only 5-core users and products are retained to ensure that each user or product is associated with at least 5 interactions. A pre-trained sentence transformer is used to extract 384-dimensional textual features, and 4096-dimensional visual features that have been published are used. The data sparsity is calculated by dividing the total number of interactions by the product of the number of products and the number of users.

[0214] The model provided in this application is compared with 11 baseline methods, which are divided into general models and multimodal models. The general model refers to a model that provides personalized recommendations to users using only the user's historical interaction sequence, while the multimodal model is a model that uses both the user's historical interaction sequence and the model information of the recommended products to make recommendations.

[0215] General models include:

[0216] 1. BPR: This is a classic collaborative filtering method that uses a matrix factorization framework to learn user and item representations.

[0217] 2. LightGCN: The most popular GCN method at present, which simplifies the GCN structure by removing feature transformations and non-linear activation functions.

[0218] Multimodal models include:

[0219] 3. VBPR: Preference prediction is performed by integrating the visual features of each item and transforming them into the visual space through linear transformation, combined with matrix factorization.

[0220] 4. MMGCN: Implemented based on graph neural networks, it builds a specific user-item bipartite graph for each modality. It achieves good recommendation results through information passing and aggregation of multi-hop neighbor information.

[0221] 5. DualGNN: Combines the specific graph of each modality with the user-related graph, and sets up a user preference module to attract the user's attention to various modalities.

[0222] 6. GRCN: A fine graph convolutional network was designed to identify false positive feedback in the interaction graph and extract corresponding noisy edges, thereby playing a pruning role and solving the implicit feedback problem.

[0223] 7. SLMRec: This method integrates self-supervised learning into a GNN-based recommendation model to learn relationships. It generates different item views through three methods and then learns the underlying training signals through comparative learning.

[0224] 8. LATTICE: Constructs a graph structure learning with modality awareness, learns the structure of objects from multimodal information, and combines multimodal information.

[0225] 9. BM3: This method simplifies the self-supervised framework, overcomes the problems of computational cost and noisy supervision signals, generates views by simply discarding edges, and incorporates graph reconstruction loss to maximize the prediction of items.

[0226] 10. MICRO: This method is an extension of LATTICE, which promotes fine-grained multimodal fusion by forcing the modality-aware representation and the multimodal fusion representation to be close.

[0227] 11. MGCN: This method proposes a novel purification mechanism that filters the original modalities of a modality and uses three modules to model user items for recommendation.

[0228] To ensure a fair comparison of each model, the model (CMC-MMR) provided in this application uses two widely used evaluation protocols for top-K recommendations: Recall@K (standard recall) and NDCG@K (normalized depreciation cumulative gain). Based on experience, K is set to 10 and 20 in this experiment. For each dataset, 80% of the historical interaction sequences are randomly divided as the training set, 10% as the validation set, and 10% as the test set.

[0229] This application implements the CMC-MMR model and all baseline methods using the MMRec framework, a unified open-source framework for developing and reproducing recommendation algorithms. To ensure fair comparison of all models, this application uniformly uses the Adam optimizer to optimize all methods and references the optimal hyperparameter settings reported in the original baseline papers. All models are implemented in PyTorch and evaluated on an NVIDIA GeForce 1660Ti GPU. For general settings, 64-dimensional Xavier initialization is used to initialize embeddings, the L2 regularization coefficient is set to 10⁻⁴, and the batch size is set to BatchSize = 2048. For the self-supervised module, a temperature coefficient τ = 0.2 is set empirically. During model training, an early stopping strategy is adopted. The total number of training iterations is 1000, and early stopping is set at 20 iterations. That is, model training is stopped after 20 iterations when the performance does not improve, with Recall@20 used as the metric for training termination.

[0230] Table 2

[0231]

[0232]

[0233] As shown in Table 2, the CMC-MMR model significantly outperforms other models, achieving average improvements of 6.202%, 4.975%, and 6.054% on the three real datasets, respectively. This indicates that the CMC-MMR model is specifically designed for multimodal recommendation. Specifically, the model first corrects and denoises the original modal data through a cross-modal correction module to prevent modal noise from affecting recommendation performance. Second, CMC-MMR constructs a product relationship graph for each modality, effectively enriching modal features while encoding the graph. For the product-user relationship graph, CMC-MMR performs degree-sensitive pruning to avoid over-smoothing of popular nodes. Finally, an adaptive preference module is constructed to mine user preferences in a self-supervised manner, rationally allocating weights for each modality to maximize the correlation between each modal feature and user behavior features. Despite the inclusion of modal data, most multimodal models are still susceptible to modal noise. For example, VBPR and MMGCN outperform LightGCN because VBPR connects visual modalities through direct links, while MMGCN extracts different modalities from the original interaction sequence for graph convolution. Both models' use of the original data leads to the propagation of modal noise during graph convolution. Although some methods, such as SLMRec with contrastive learning and MGCN with modality cleansing, attempt to address this issue, they do not fully utilize the relationships between modalities. In contrast, the CMC-MMR model, through cross-modal correction, not only eliminates noise but also fully utilizes the mutual information between modalities. Therefore, the CMC-MMR model achieves the best recommendation performance.

[0234] It can be seen that using multimodal data for recommendations often yields better results than using a single modality. GRCN, SLMRec, and LATTICE are all multimodal models. For example, VBPR introduces visual patterns, significantly improving the performance of BPR-based recommendations. However, the success of multimodal recommendations depends on the quality of the modal features, and this quality can vary across different datasets. For instance, DualGCN, which utilizes a bi-graph structure, outperforms LightGCN on clothing datasets, but performs poorly on infant and sports datasets.

[0235] In addition, processing the original modalities and introducing other methods can significantly improve the performance of multimodal recommendations. For example, GRCN introduces an attention mechanism on the basis of the original modalities, and uses the attention mechanism to prune products that have little impact on users; the MGCN model uses a behavior-guided purifier to purify the original modalities, and the final recommendation effect is second only to the model proposed in this paper.

[0236] Furthermore, to verify the impact of each module in the model provided in this application on the recommendation results, the embodiments of this application also conducted ablation experiments on the CMC-MMR model. Specifically, the embodiments of this application set up three variants of the model:

[0237] (1) w / o MF: The cross-modal correction module has been removed from the original model. Therefore, in this model, the original modal data will be directly entered into the model without any processing.

[0238] (2) w / o DP: Degree-sensitive pruning was removed from the original model, so that the product user relationship graph enters the view encoder module without any pruning.

[0239] (3) w / o SS: In this model, the self-supervised task has been removed, and naturally, the SS task has also been removed. The loss, and the final product feature representation, is obtained by averaging the representations of the product relationship graph convolution and the product user relationship graph convolution.

[0240] like Figure 6 The diagram shown illustrates the test results of the ablation experiment described above. Figure 6 It can be seen from this:

[0241] (1) Compared with the CMC-MMR model, all three variant models showed poor recommendation performance, which indicates that any module of the CMC-MMR model is essential. Among them, the cross-modal correction module has the greatest impact on the Baby dataset, which shows the effectiveness of the cross-modal correction module. It can greatly remove the noise of the original modality, thereby improving the recommendation performance. The impact on the other two datasets is second only to deletion-sensitive pruning. We believe this is mainly due to the different scale of the datasets. Compared with the Sports and Clothing datasets, the Baby dataset has fewer user product interactions, so the model's recommendation depends more on the different modalities of the product. Therefore, whether or not the modality is corrected has the greatest impact on the Baby dataset.

[0242] To better explain how the cross-modal correction module can effectively remove noise from the original modality, this application also visualizes the representation distribution of the Clothing dataset. First, 500 items are randomly selected from the Clothing dataset, and their textual and visual modal features are extracted. The modal features are mapped to a two-dimensional space using tSNE. Then, the 2D feature distribution is plotted using Gaussian kernel density estimation (KDE). To more clearly present the corrected results, the arctan(y,x) of each graph is visualized on a unit hypersphere S. 1 Gaussian kernel density estimation on [aspect name]. Experimental results for the visual modality are as follows: Figure 7 and Figure 8 As shown, Figure 7 This is a schematic diagram of the visual modality feature distribution before correction. Figure 8 The diagram shows the corrected visual modality feature distribution. The experimental results for the text modality are as follows: Figure 9 and Figure 10 As shown, Figure 9 This is a schematic diagram of the modal feature distribution of the text before correction. Figure 10 This is a schematic diagram of the modal feature distribution of the corrected text.

[0243] The distribution of 2D features reveals that the original modal features exhibit multiple community clusters and a non-uniform unimodal pattern, indicating the complexity of user-product relationships within the original modal features and also suggesting low discriminative power, which will negatively impact the performance of the recommendation system. Furthermore, the distribution of both text and image modal features is highly uneven, leading to a decrease in accuracy when using a single modality for product recommendations. After cross-modal information correction, the 2D feature distribution becomes more uniform, removing noise from the original modal features and increasing their discriminative power.

[0244] (2) Degree-sensitive pruning strategy plays a crucial role in improving the model, especially in datasets with a large number of interactions, such as Sports and Clothing. In datasets with a large number of interactions, there are a certain number of noisy edges. Degree-sensitive pruning can reasonably remove noisy edges, thereby maximizing the interaction that is beneficial to the model during graph convolution and enriching the user's behavioral features.

[0245] (3) The adaptive preference module has the least influence compared to the other two modules, but it is still significantly lower than the CMC-MMR model. This shows that the adaptive preference module still plays a very important role. It also shows that the adaptive preference module can adapt to the user's preferences and assign different weights to each modality.

[0246] In the embodiments of this application, CMC-MMR uses text modality and image modality. In order to verify that the CMC-MMR model is designed for the multimodal domain, this application also explores the impact of each modality on the model. The specific results are shown in Table 3, where Text indicates that only the text modality is used and Image indicates that only the image modality is used. In the model using a single modality, each modality is corrected only by the ID embedding vector.

[0247] Table 3

[0248] Baby Text 0.0645 0.1013 0.0344 0.0438 Image 0.0551 0.0862 0.0295 0.0375 All 0.0664 0.1042 0.0351 0.0448 Sports Text 0.0540 0.0811 0.0297 0.0365 Image 0.0603 0.0910 0.0334 0.0413 All 0.0770 0.1148 0.0421 0.0518 Clothing Text 0.0630 0.0935 0.0344 0.0422 Image 0.0444 0.0674 0.0240 0.0299 All 0.0691 0.1001 0.0368 0.0447

[0249] According to the data in Table 3, it can be found that the recommendation performance of using single-modal features is significantly lower than that of using multimodal features. In most cases, text modal features are better than image modal features. This is because using a single image modal and only correcting with the product ID embedding vector without cross-modal correction of the text modal results in insignificant noise removal of the original image modal and loss of relevant information between modalities. Another reason is that users may prefer text modal when selecting products compared to image modal.

[0250] The CMC-MMR multimodal product recommendation model provided in this application processes the raw information through three core modules: First, a cross-modal correction module removes noise from the raw modal data while maximizing the relevant information between different modalities. Second, a view encoder module convolves the product relationship graph and the product-user relationship graph to enrich product features and mine user features. Finally, an adaptive preference module is designed to adaptively mine user preferences for each modality, thereby assisting in improving the performance of the recommendation system.

[0251] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0253] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0254] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0255] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal product recommendation method based on cross-modal correction, characterized in that, include: Obtain a product dataset containing text information, image information, and product ID vectors for multiple products, and construct a product relationship graph based on the product dataset; Obtain a user dataset containing historical interaction data and user ID vectors of multiple users, and construct a product user relationship graph based on the product dataset and the user dataset; The product ID vector, image information, and text information of each product in the product dataset are input into the cross-modal correction module of the trained multimodal product recommendation model. The model outputs the target image feature vector and target text feature vector for each product. Based on these target image and text feature vectors, the modal feature vector for each product is obtained, specifically including: The MLP neural network in the cross-modal correction module is used to extract features from the image and text information of each product, outputting high-order image feature vectors and high-order text feature vectors for each product; wherein, the high-order image feature vectors for each product are: ， in, Indicates the first High-order image feature vectors of each product This is the first weight matrix. This is the first bias matrix. Indicates the first Image information of each product; The high-order text feature vectors for each product are as follows: ， in, Indicates the first The high-order textual feature vector of each product Indicates the first Text information for each product; The high-order text feature vector of each product is corrected using its product ID vector to obtain the target text feature vector of each product. Then, the high-order image feature vector of each product is corrected using its target text feature vector to obtain the target image feature vector of each product. The target text feature vector of each product is as follows: ， in, Indicates the first The target text feature vector of each product Indicates the first Product ID vector for each product Represents element-wise product. It is a sigmoid non-linear activation function. This is the second weight matrix. This is the second bias matrix; The feature vectors of the target images for each product are as follows: ， in, Indicates the first The feature vector of the target image of a product Represents element-wise product. It is a sigmoid non-linear activation function. This is the third weight matrix. This is the third bias matrix; The modal feature vectors of each product are obtained by combining the target text feature vector and the target image feature vector of each product. The modal feature vectors of each product and the product relationship graph are input into the view encoder module of the trained multimodal product recommendation model, and the target modal feature vectors of each product are output; the product user relationship graph is input into the view encoder module, and the product feature representations of each product and the user feature representations of each user are output; The adaptive preference module in the trained multimodal product recommendation model is used to extract the user behavior feature vector of each user's user ID vector and the shared modal feature vector of each product's target modal feature vector. The user behavior feature vector, the shared modal feature vector of each product, and the target modal feature vector of each product are input into the adaptive preference module, and the modal preference feature representation of each user for each product is output. Based on the product feature representation and modal preference feature representation of each product, the product target feature representation of each product is obtained. Based on the user feature representation and modal preference feature representation of each user, the user target feature representation of each user is obtained. Based on the user target feature representation and the product target feature representation, the preference score between the user and the product is calculated so as to recommend products to the user based on the preference score.

2. The multimodal product recommendation method based on cross-modal correction according to claim 1, characterized in that, The modal feature vectors of each product and the product relationship graph are input into the view encoder module of the trained multimodal product recommendation model, and the target modal feature vectors of each product are output, including: Calculate the similarity between each product and its neighboring products in the product relationship graph, and perform KNN sparsification on the product relationship graph. Based on the similarity between each product and its neighboring products after sparsification, obtain the similarity matrix of the product relationship graph. Based on the similarity matrix and the modal feature vector of each product, the product modal feature vector of each product is obtained, and the modal preference feature vector of the user is calculated based on the product modal feature vector of each product. The target modal feature vector of each product is obtained by concatenating the product modal feature vector and the user modal preference feature vector.

3. The multimodal product recommendation method based on cross-modal correction according to claim 2, characterized in that, The formula for calculating the similarity between each product and its adjacent products in the product relationship diagram is as follows: ， in, In the product relationship diagram, the first... The similarity between a product and its adjacent products Indicates the relationship between the product relationship diagram and the first Products adjacent to each other Indicates the first Modal feature vectors of each product Indicates the first Modal feature vectors of each product Represents the transpose of a matrix; The formula for calculating the product modal feature vector of each product is as follows: ， in, Indicates the first Product modal feature vector of each product This indicates that the product relationship diagram is in modal The similarity matrix below; The formula for calculating the user's modal preference feature vector is: ， in, Indicates the first Modal preference feature vector of each user Indicates user The collection of products that have been interacted with Indicates relation to product A collection of users who have interacted with the user.

4. The multimodal product recommendation method based on cross-modal correction according to claim 1, characterized in that, The product user relationship diagram is input into the view encoder module, and the output includes product feature representations of each product and user feature representations of each user, including: The probability of each connection line in the product user relationship graph being pruned is calculated based on the degree-sensitive probability pruning strategy. The product user relationship graph is pruned based on the probability of each connection line being pruned and the preset pruning ratio. A symmetric adjacency matrix is ​​constructed based on the pruned product user relationship graph. The view encoder module is used to perform graph convolution on the symmetric adjacency matrix to output the product feature representation of each product and the user feature representation of each user. The formula for calculating the probability of each connecting line in the product user relationship diagram being pruned is as follows: ， in, This represents the product user relationship diagram. The probability that a connecting line will be trimmed. Represents a node In the product user relationship diagram, the degree Represents a node Degrees and nodes in a product user relationship graph and nodes For the first The two nodes of the connecting line; The product characteristics of each product are represented as follows: ， in, Indicates the first Product characteristics representation for each product This indicates the number of layers in the graph convolution operation. Indicates the first The output of layer graph convolution operation; ， in, Indicates user The collection of products that have been interacted with Indicates relation to product The set of users who have interacted with each other. Indicates the first The output of the layer graph convolution operation is the first User characteristics representation of each user; The user characteristics of each user are represented as follows: ， in, Indicates the first User characteristics representation of each user Indicates the first The output of layer graph convolution operation; ， in, Indicates the first The output of the layer graph convolution operation is the first Product characteristics representation for each product.

5. The multimodal product recommendation method based on cross-modal correction according to claim 1, characterized in that, The adaptive preference module in the trained multimodal product recommendation model extracts the user behavior feature vector from the user ID vector of each user and the shared modal feature vector from the target modal feature vector of each product. The user behavior feature vector, the shared modal feature vector of each product, and the target modal feature vector of each product are then input into the adaptive preference module, outputting a modal preference feature representation for each user towards each product, including: The feature extraction submodule in the adaptive preference module is used to extract the behavioral feature vector of each user's user ID vector; The attention submodule in the adaptive preference module is used to extract the shared modal feature vector of the target modal feature vector of each product; The behavioral feature vector of each user, the shared modal feature vector of each product, and the target modal feature vector of each product are input into the feature fusion submodule in the adaptive preference module, and the modal preference feature representation of each user for each product is output.

6. The multimodal product recommendation method based on cross-modal correction according to claim 5, characterized in that, The behavioral feature vectors of each user are as follows: ， in, Indicates the first A user's behavioral feature vector It is a sigmoid non-linear activation function. This is the fourth weight matrix. Indicates the first A user ID vector for each user. This is the fourth bias matrix; The shared modal feature vectors of all products are: ， in, Indicates the first Shared modal feature vectors of individual products Represents a set of modes. Indicates the first Attention score of the target modal feature vector of a product. Indicates the first The target modal feature vector of each product; ， in, This is the fifth weight matrix. This is the fifth bias matrix. For attention vectors, and It is a non-linear activation function; Each user's modal preference characteristics for each product are represented as follows: ， in, Indicates containing the first The user on the first Modal preference features of each product This represents the product of elements.

7. The multimodal product recommendation method based on cross-modal correction according to claim 6, characterized in that the product target features of each product are represented as follows: ， in, Indicates the first Product target characteristics representation for each product. Indicates the first Product characteristics representation for each product The modality preference feature representation in the first part Product description for each product; The user's target characteristics are represented as follows: ， in, Indicates the first User target characteristics representation for each user Indicates the first User characteristics representation of each user The modality preference feature representation in the first part User representation for each user; The formula for calculating the user's preference score for the product is: ， in, Indicates the first The user on the first Preference scores for each product This represents the transpose of a matrix.

8. The multimodal product recommendation method based on cross-modal correction according to claim 7, characterized in that, The training process of a multimodal product recommendation model includes: Multiple triples containing positive and negative samples were constructed based on the product and user datasets in the training set. Multiple triples are input into the multimodal product recommendation model for iterative training until the product recommendation loss function is minimized, thus obtaining the trained multimodal product recommendation model. Among them, the triplet , Represents the first in the user dataset One user, Represents users in the product dataset Clicked the One product Represents users in the product dataset Unclicked One product; The product recommendation loss function is: ， in, This represents the product recommendation loss function. Indicates L2 regularization, Represents the BPR loss function. This represents the loss function for a self-supervised task. As the first hyperparameter, This is the second hyperparameter; BPR loss function for: ， in, Indicates the first The user on the first Preference scores for each product Indicates the first The user on the first Preference scores for each product; Self-supervised task loss function for: ， in, Represents the user dataset. Represents the product dataset. Represents the first in the user dataset The first characteristic of a user is represented by Represents the first in the user dataset The second characteristic of each user is represented by This represents the temperature hyperparameter of the softmax function. Indicates the first product in the dataset The first characteristic of a product is represented by Indicates the first product in the dataset The second characteristic of a product is represented.

Images