Multimodal recommendation method and system for enhancing user representation using graph convolutional neural network

Abstract

The present invention discloses a multimodal recommendation method and system for enhancing user representation with graph convolutional neural network. The present invention includes extracting project embedding, original user embedding and original ID embedding from the input data set, extracting features from the user's neighbors and performing aggregation operations to enhance the user representation to generate a user embedding after aggregating other user information, and then splicing it with the project embedding into a user-project collaborative interaction matrix, extracting features of each modality for the input data set and building a user-item graph of each modality in combination with the user-project collaborative interaction matrix; and then obtaining the user's recommended items through multimodal information representation, multimodal feature fusion and recommendation results. The present invention aims to improve the accuracy of recommendations by introducing multimodal information while introducing user feature enhancement technology, obtain the user's personalized embedding by aggregating the user features with the features of similar users, and finally improve the recommendation accuracy of the multimodal recommendation system.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and recommendation systems, and in particular to a multimodal recommendation method and system for enhancing user representation using a graph convolutional neural network. Background Art

[0002] Early recommendation technologies mainly relied on a single type of information, such as the user's past interactions, and failed to fully utilize the multimodal features of the items, so they could not fully capture the user's preference signals. With the advancement of deep learning technology, recommendation systems can now extract deep features and complex nonlinear relationships in multimodal data through neural networks, thus developing multimodal recommendation systems. Multimodal recommendation systems combine user behavior data with multimodal information of items, and by analyzing the relationship between different data sources, they can more comprehensively understand the user's interests and preferences, and then provide more accurate personalized recommendations. Multimodal information enriches the feature representation of items and partially alleviates the data sparsity problem, thereby effectively improving the performance of the recommendation model. However, this also brings challenges. First, the shared update of user ID embedding may lead to coupling between collaborative signals and multimodal signals. Second, due to the differences in semantic information of different modalities, it is necessary to design a reasonable feature alignment method to enhance the semantic consistency between modalities. Finally, a reasonable multimodal feature fusion mechanism is also crucial to ensure that each modality plays an active role in the recommendation process. Effective feature alignment and fusion methods can not only reduce the training cost of the model, but also significantly improve the performance of the model. How to solve the above challenges of multimodal recommendation systems has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multimodal recommendation method and system with enhanced user representation by a graph convolutional neural network is provided. The present invention aims to improve the accuracy of recommendations by introducing multimodal information while introducing user feature enhancement technology, and to obtain personalized embedding of users by aggregating user features with features of similar users, thereby ultimately improving the recommendation accuracy of the multimodal recommendation system.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A multimodal recommendation method for enhancing user representation using a graph convolutional neural network comprises the following steps:

[0006] S1, multimodal data preprocessing, including: extracting item embeddings from the input dataset , original user embedding and the original ID embedded , extracting features from the user’s neighbors and performing aggregation operations to enhance the user representation to generate a user embedding that aggregates other users’ information , and then embed it with the project Splicing into a user-item collaborative interaction matrix , extract the features of each modality for the input data set and combine it with the user-item collaborative interaction matrix Constructing user-item graphs for each modality ,in Indicates modality;

[0007] S2, multimodal information representation, including: embedding the original ID into Use graph convolutional neural network to extract high-order features to get ID embedding graph ; The user-item graph of each modality Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality ;

[0008] S3, multimodal feature fusion, including: embedding ID into graph And the modal embedding diagram of each mode Fusion gets the final embedding graph ;

[0009] S4, recommendation result prediction, including: based on the final embedding graph Split to get the user part With project part , the user part With project part according to Perform inner product calculation to get user About Project Final prediction score ,in Indicates transposition, according to the user About Project Final prediction score Select the first K items to display to the user The K recommended items.

[0010] Optionally, step S1 includes:

[0011] S1.1, Extract item embeddings from the input dataset , original user embedding and the original ID embedded ;

[0012] S1.2, obtain the first K items that each user interacts with from the input user-item interaction graph. If there are less than K interactive items, randomly select items from the interactive items as supplements, extract the IDs of the first K items and the weights between the items, and obtain the ID list of the first K items that the user interacts with. And the weight matrix ;

[0013] S1.3, embedding the original user Combined with the ID list of the first K items that the user interacted with And the weight matrix , using the isomorphic user co-occurrence graph to learn the user embedding after aggregating other users’ information ;

[0014] S1.4, embedding users with other users’ information and project embedding Splicing into a user-item collaborative interaction matrix ;

[0015] S1.5, for the input data set, use the specified image feature extraction network model to extract visual features, and use the text feature extraction network model to extract text description information and id text features;

[0016] S1.6, the user-item collaborative interaction matrix The users and items in the graph are represented as nodes respectively. The connection relationship between users and items is constructed based on the user's behavioral association with the items. The visual features and text features are associated with the nodes corresponding to the users and items respectively, thus forming the user-item graph of each modality. ,in Indicates the mode, , Represents visual, text and ID modes respectively.

[0017] Optionally, in step S1.3, the function expression of the user embedding obtained by using the isomorphic user co-occurrence graph to learn and aggregate other user information is:

[0018] ,

[0019] In the above formula, Represents the user embedding after aggregating other users’ information. Indicates user Other users, represents the weight matrix of the first K items that the user interacted with, represents the original user embedding, A list of ids of the top K items that the user has interacted with.

[0020] Optionally, the user-item collaborative interaction matrix obtained by splicing in step S1.4 is The function expression is:

[0021] ,

[0022] In the above formula, represents the user-item collaborative interaction matrix, Indicates row-by-row concatenation operation. represents user embeddings that aggregate other users’ information, Indicates that the item is embedded.

[0023] Optionally, in step S2, the original ID is embedded in Use graph convolutional neural network to extract high-order features to get ID embedding graph When, the graph convolutional neural network includes Layer, original ID embedding The ID embedding for the input of layer 1 , and the function expression of the graph propagation function used in each layer is:

[0024] ,

[0025] In the above formula, Indicates The ID embedding of the layer output, Indicates The ID embedding of the layer output, represents the graph propagation function, Represents the user-item collaborative interaction matrix The constructed adjacency matrix is for The diagonal matrix of , and finally the ID embedding graph is obtained The function expression is:

[0026] ,

[0027] In the above formula, It means taking the average value, ~ They are No. 1~ The output of the layer is the ID embedding.

[0028] Optionally, in step S2, the user-item graphs of each modality are Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality include:

[0029] S2.1, User-Item Graphs from Each Modality The high-level features are captured and projected into a unified embedding space:

[0030] ,

[0031] In the above formula, Display items No. The projection features of the modes, Indicates The modal transformation matrix of the modes is Display items No. The projection characteristics of each mode;

[0032] S2.2, the user modal features are obtained by aggregating the projection features according to the following formula:

[0033] ,

[0034] In the above formula, For users No. The user modality features of the modality, For users In the user-item collaborative interaction matrix The set of neighbors in ;

[0035] S2.3, construct the modal characteristic matrix of each mode according to the following formula:

[0036] ,

[0037] In the above formula, Indicates The modal characteristic matrix of the modes, ~ Respectively represent the 1st to User's The user modality features of the modality, is the number of users, ~ Respectively represent the 1st to Project No. The projection features of the modes, is the number of items;

[0038] S2.4, extracting high-order features from the modal feature matrix of each modality using a graph convolutional neural network to obtain a modal embedding graph of each modality, wherein the graph convolutional neural network includes Layer, the modal feature matrix of each modality is the modal embedding of the input of the first layer , and the function expression of the graph propagation function used in each layer is:

[0039] ,

[0040] In the above formula, Indicates The output of the layer modality embedding of modalities, Indicates The output of the layer The modal embedding of each modality is performed The modal embedding combination of the modalities is obtained The modal embedding graph of the modal .

[0041] Optionally, in step S3, the ID is embedded in the image And the modal embedding diagram of each mode Fusion gets the final embedding graph The function expression is:

[0042] ,

[0043] In the above formula, represents collaborative embedding, Indicates The modal embedding graph of the modes, represents the normalization function used to alleviate the difference in value scale between modal embedding maps, is the number of modes.

[0044] In addition, the present invention also provides a multimodal recommendation system with enhanced user representation using a graph convolutional neural network, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal recommendation method with enhanced user representation using a graph convolutional neural network.

[0045] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal recommendation method of enhancing user representation by using a graph convolutional neural network through a processor.

[0046] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal recommendation method of enhancing user representation by using a graph convolutional neural network through a processor.

[0047] Compared with the prior art, the present invention mainly has the following advantages:

[0048] 1. The present invention enriches the feature representation of projects and users by introducing multimodal information, and captures the high-order features of each modality through a graph convolutional network, which can more accurately extract user preference clues and achieve better personalized recommendations.

[0049] 2. The present invention introduces a method for constructing user features based on user co-occurrence graphs, which learns the features of users with similar item interactions to enrich user features.

[0050] 3. The present invention introduces a graph embedding-based method, which embeds the user-project collaboration graph through operations such as convolution and aggregation on the collaboration graph information between users and projects.

[0051] 4. The present invention introduces a method for fusing multimodal information with user-project collaborative information. Multimodal embedding and collaborative graph embedding are projected into a space through a projection function and then aggregated to enhance the semantic consistency between different modalities, thereby achieving the effect of improving the recommendation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention relates to the technical field of data processing and recommendation systems, and specifically uses a graph convolutional neural network to learn collaborative correlation graphs and modal embedding graphs of items, thereby obtaining user interests and achieving more accurate personalized recommendations. The present invention will be further described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, the multimodal recommendation method for enhancing user representation using a graph convolutional neural network in this embodiment includes the following steps:

[0054] S1, multimodal data preprocessing, including: extracting item embeddings from the input dataset , original user embedding and the original ID embedded , extracting features from the user’s neighbors and performing aggregation operations to enhance the user representation to generate a user embedding that aggregates other users’ information , and then embed it with the project Splicing into a user-item collaborative interaction matrix , extract the features of each modality for the input data set and combine it with the user-item (UI) collaborative interaction matrix Constructing user-item graphs for each modality ,in Indicates modality;

[0055] S2, multimodal information representation, including: embedding the original ID into Use graph convolutional neural network to extract high-order features to get ID embedding graph ; The user-item graph of each modality Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality ;

[0056] S3, multimodal feature fusion, including: embedding ID into graph And the modal embedding diagram of each mode Fusion gets the final embedding graph ;

[0057] S4, recommendation result prediction, including: based on the final embedding graph Split to get the user part With project part , the user part With project part according to Perform inner product calculation to get user About Project Final prediction score ,in Indicates transposition, according to the user About Project Final prediction score Select the first K items to display to the user The K recommended items.

[0058] In this embodiment, step S1 includes:

[0059] S1.1, Extract item embeddings from the input dataset , original user embedding and the original ID embedded In this embodiment, the MMRec built-in tool class is used to extract the project embedding in the data set. , original user embedding and the original collaborative embedding ,In addition, other methods or tools can be used as needed;

[0060] S1.2, obtain the first K items that each user interacts with from the input user-item interaction graph. If there are less than K interactive items, randomly select items from the interactive items as supplements, extract the IDs of the first K items and the weights between the items, and obtain the ID list of the first K items that the user interacts with. And the weight matrix ;

[0061] S1.3, embedding the original user Combined with the ID list of the first K items that the user interacted with And the weight matrix , using the isomorphic user co-occurrence graph to learn the user embedding after aggregating other users’ information ;

[0062] S1.4, embedding users with other users’ information and project embedding Splicing into a user-item collaborative interaction matrix ;

[0063] S1.5, for the input data set, use the specified image feature extraction network model to extract visual features, and use the text feature extraction network model to extract text description information and text features of the id; as an optional implementation, in this embodiment, the pre-trained ResNet50 model is used to extract image features corresponding to the image information in the data set, and the pre-trained SentenceTransformer model is used to extract text features corresponding to the text description information and the id in the data set; in addition, other image feature extraction network models and text feature extraction network models can also be used as needed;

[0064] S1.6, the user-item collaborative interaction matrix The users and items in the graph are represented as nodes respectively. The connection relationship between users and items is constructed based on the user's behavioral association with the items. The visual features and text features are associated with the nodes corresponding to the users and items respectively, thus forming the user-item graph of each modality. ,in Indicates the mode, , Represents visual, text and ID modes respectively.

[0065] In this embodiment, the function expression of the user embedding obtained by using the isomorphic user co-occurrence graph to learn and aggregate other user information in step S1.3 is:

[0066] ,

[0067] In the above formula, Represents the user embedding after aggregating other users’ information. Indicates user Other users, represents the weight matrix of the first K items that the user interacted with, represents the original user embedding, A list of ids of the top K items that the user has interacted with.

[0068] In this embodiment, the user-item collaborative interaction matrix is ​​obtained by splicing in step S1.4 The function expression is:

[0069] ,

[0070] In the above formula, represents the user-item collaborative interaction matrix, Indicates row-by-row concatenation operation. represents user embeddings that aggregate other users’ information, Indicates that the item is embedded.

[0071] Embed the original ID The high-order features can be extracted by using graph convolutional neural network and the message propagation on the graph can be obtained to capture the high-order connectivity. In step S2 of this embodiment, the original ID is embedded into Use graph convolutional neural network to extract high-order features to get ID embedding graph When, the graph convolutional neural network includes Layer, original ID embedding The ID embedding for the input of layer 1 , and the function expression of the graph propagation function used in each layer is:

[0072] ,

[0073] In the above formula, Indicates The ID embedding of the layer output, Indicates The ID embedding of the layer output, represents the graph propagation function, Represents the user-item collaborative interaction matrix The constructed adjacency matrix is for The diagonal matrix of , and finally the ID embedding graph is obtained The function expression is:

[0074] ,

[0075] In the above formula, It means taking the average value, ~ They are No. 1~ The output of the layer is the ID embedding.

[0076] In this embodiment, in step S2, the user-item graphs of each modality are Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality include:

[0077] S2.1, User-Item Graphs from Each Modality The high-level features are captured and projected into a unified embedding space:

[0078] ,

[0079] In the above formula, Display items No. The projection features of the modes, Indicates The modal transformation matrix of the modes is Display items No. The projection characteristics of each mode;

[0080] S2.2, the user modal features are obtained by aggregating the projection features according to the following formula:

[0081] ,

[0082] In the above formula, For users No. The user modality features of the modality, For users In the user-item collaborative interaction matrix The set of neighbors in ;

[0083] S2.3, construct the modal characteristic matrix of each mode according to the following formula:

[0084] ,

[0085] In the above formula, Indicates The modal characteristic matrix of the modes, ~ Respectively represent the 1st to User's The user modality features of the modality, is the number of users, ~ Respectively represent the 1st to Project No. The projection features of the modes, is the number of items;

[0086] S2.4, extracting high-order features from the modal feature matrix of each modality using a graph convolutional neural network to obtain a modal embedding graph of each modality, wherein the graph convolutional neural network includes Layer, the modal feature matrix of each modality is the modal embedding of the input of the first layer , and the function expression of the graph propagation function used in each layer is:

[0087] ,

[0088] In the above formula, Indicates The output of the layer modality embedding of modalities, Indicates The output of the layer The modal embedding of each modality is performed The modal embedding combination of the modalities is obtained The modal embedding graph of the modal In this embodiment, the two graph convolutional neural networks used in step S2 are independent graph convolutional neural networks, and when training the graph convolutional neural network, the following Bayesian personalized ranking loss is used as the basic optimization function in this embodiment:

[0089] ,

[0090] In the above formula, represents the Bayesian personalized ranking loss, represents the input data set, represents the logarithmic function, is the sigmoid activation function, Indicates user About Project The final prediction score of Indicates user About Project The final prediction score of For users Projects that have interacted, For users No interactive items have been generated.

[0091] In this embodiment, in step S3, the ID is embedded in the image. And the modal embedding diagram of each mode Fusion gets the final embedding graph The function expression is:

[0092] ,

[0093] In the above formula, represents collaborative embedding, Indicates The modal embedding graph of the modes, represents the normalization function used to alleviate the difference in value scale between modal embedding maps, is the number of modes.

[0094] In order to verify the multimodal recommendation method of enhancing user representation with graph convolutional neural network in this embodiment, this embodiment uses Amazon product review datasets, including Baby dataset, Sports dataset and Clothing dataset, in the experiment, uses recall rate (Recall) and normalized discounted cumulative gain (NDCG), and uses the existing multimodal recommendation model LATTICE, multimodal graph contrastive learning recommendation model MMGCL, multimodal recommendation model MICRO, self-supervised learning guided multimedia recommendation model SLMREC and multimodal recommendation model BM3 as comparisons of the method of this embodiment (EURec). The final experimental results are shown in Tables 1, 2 and 3.

[0095] Table 1: Experimental results comparing the performance of other recommendation methods on the Baby dataset

[0096]

[0097] Table 2: Experimental results comparing the performance of other recommendation methods on the Sports dataset

[0098]

[0099] Table 3: Experimental results comparing the performance of other recommendation methods on the Clothing dataset

[0100]

[0101] In Table 1, Table 2 and Table 3, R@10 represents the recall rate when the recommended item K is 10, R@20 represents the recall rate when the recommended item K is 20, N@10 represents the normalized discounted cumulative gain when the recommended item K is 10, and N@20 represents the normalized discounted cumulative gain when the recommended item K is 20. Referring to Table 1, Table 2 and Table 3, it can be seen that the method (EURec) of this embodiment has better recall rate and normalized discounted cumulative gain than the existing multimodal recommendation model LATTICE, multimodal graph contrastive learning recommendation model MMGCL, multimodal recommendation model MICRO, multimedia recommendation model guided by self-supervised learning SLMREC and multimodal recommendation model BM3 under various recommended item K of 10 and 20 and three data sets of Baby data set, Sports data set and Clothing data set, and has the advantage of high prediction accuracy.

[0102] In summary, the multimodal recommendation system introduces user feature enhancement technology while improving the accuracy of recommendations by introducing multimodal information, but this also brings new challenges. The multimodal recommendation system uses multiple modal information to simulate the user's preference for items, and the representation of the same item in different modalities may be different, which may lead to a decrease in the accuracy of the final prediction result. To this end, the multimodal recommendation method of the graph convolutional neural network in this embodiment to enhance user representation includes first processing the interaction data between the user and the item, based on the user feature enhancement graph structure, extracting features from the user's neighbors and performing aggregation operations, thereby generating a personalized embedding of the user, and splicing it with the project embedding to obtain a user-item (UI) collaborative interaction graph. Then, the interaction graph is subjected to a graph convolution operation and the multi-layer embedding is aggregated to obtain the embedding of the collaborative graph. Secondly, the multimodal content of the project is incorporated. The multimodal user feature vector is obtained according to the multimodal matrix and the project-image conversion matrix, and is connected to the user feature vector. The new feature matrix is ​​subjected to multiple graph convolution operations, and after completion, the image and text features are normalized and added to obtain the modal graph embedding. The above-mentioned collaborative graph embedding and modal graph embedding are then added to obtain the final embedding representation. This embodiment also includes a Bayesian personalized ranking loss optimization function, which is used as the basis for optimizing the model. The trained model can make personalized recommendations to users based on the user's historical interaction behavior and multimodal information. The multimodal recommendation method for enhancing user representation using a graph convolutional neural network in this embodiment introduces user feature enhancement technology while introducing multimodal information to improve the accuracy of recommendations. The user feature enhancement technology of this embodiment is implemented by extracting features from the user's neighbors and performing aggregation operations to enhance the user representation to generate a user embedding after aggregating other user information. , the user features are aggregated with the features of similar users to obtain the user's personalized embedding, so as to independently learn the collaboratively related and modally related user and item representations, thereby avoiding unstable updates of user embeddings and promoting decoupled learning of user interests. At the same time, the user features are aggregated with the features of similar users to obtain the user's personalized embedding. This technology can enrich user feature information so that the recommendation system can provide more accurate personalized recommendations, thereby improving user experience and satisfaction, improving the accuracy and personalization level of the recommendation system, and ultimately improving the recommendation accuracy of the multimodal recommendation system. This embodiment effectively improves the prediction accuracy of the recommendation system by combining multimodal information. By fusing the collaborative embedding of users and items with multimodal embedding, this technology can provide users with more accurate personalized recommendations and is suitable for various multimodal recommendation application scenarios.

[0103] In addition, this embodiment also provides a multimodal recommendation system with enhanced user representation using a graph convolutional neural network, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal recommendation method with enhanced user representation using a graph convolutional neural network.

[0104] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal recommendation method of enhancing user representation by using a graph convolutional neural network through a processor.

[0105] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal recommendation method for enhancing user representation by using a graph convolutional neural network through a processor.

[0106] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal recommendation method for enhancing user representation using a graph convolutional neural network, characterized in that: The steps include: S1, multimodal data preprocessing, including: extracting item embeddings from the input dataset , Original User Embed and the original ID embedded , extracting features from the user’s neighbors and performing aggregation operations to enhance the user representation to generate a user embedding that aggregates other users’ information , and then embed it with the project Splicing into a user-item collaborative interaction matrix , extract the features of each modality for the input data set and combine it with the user-item collaborative interaction matrix Constructing user-item graphs for each modality ,in Indicates modality; S2, multimodal information representation, including: embedding the original ID into Use graph convolutional neural network to extract high-order features to get ID embedding graph ; The user-item graph of each modality Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality ; S3, multimodal feature fusion, including: embedding ID into graph And the modal embedding diagram of each mode Fusion gets the final embedding graph ; S4, recommendation result prediction, including: based on the final embedding graph Split to get the user part With project part , the user part With project part according to Perform inner product calculation to get user About Project Final prediction score ,in Indicates transposition, according to the user About Project Final prediction score Select the first K items to display to the user The K recommended items.

2. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 1, characterized in that: Step S1 includes: S1.1, Extract item embeddings from the input dataset , Original User Embed and the original ID embedded ; S1.2, obtain the first K items that each user interacts with from the input user-item interaction graph. If there are less than K interactive items, randomly select items from the interactive items as supplements, extract the IDs of the first K items and the weights between the items, and obtain the ID list of the first K items that the user interacts with. And the weight matrix ; S1.3, embedding the original user Combined with the ID list of the first K items that the user interacted with And the weight matrix , using the isomorphic user co-occurrence graph to learn the user embedding after aggregating other users’ information ; S1.4, embedding users with other users’ information and project embedding Splicing into a user-item collaborative interaction matrix ; S1.5, for the input data set, use the specified image feature extraction network model to extract visual features, and use the text feature extraction network model to extract text description information and id text features; S1.6, the user-item collaborative interaction matrix The users and items in the graph are represented as nodes respectively. The connection relationship between users and items is constructed based on the user's behavioral association with the items. The visual features and text features are associated with the nodes corresponding to the users and items respectively, thus forming the user-item graph of each modality. ,in Indicates the mode, , Represents visual, text and ID modes respectively.

3. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 2, characterized in that: In step S1.3, the function expression of the user embedding after aggregating other user information is obtained by using the isomorphic user co-occurrence graph to learn is: ， In the above formula, Represents the user embedding after aggregating other users’ information. Indicates user Other users, represents the weight matrix of the first K items that the user interacted with, represents the original user embedding, A list of ids of the top K items that the user has interacted with.

4. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 2, characterized in that: The user-item collaborative interaction matrix is ​​obtained by splicing in step S1.4 The function expression is: ， In the above formula, represents the user-item collaborative interaction matrix, Indicates row-by-row concatenation operation. represents user embeddings that aggregate other users’ information, Indicates that the item is embedded.

5. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 1, characterized in that: In step S2, the original ID is embedded into Use graph convolutional neural network to extract high-order features to get ID embedding graph When, the graph convolutional neural network includes Layer, original ID embedding The ID embedding for the first layer input , and the function expression of the graph propagation function used in each layer is: , In the above formula, Indicates The ID embedding of the layer output, Indicates The ID embedding of the layer output, represents the graph propagation function, Represents the user-item collaborative interaction matrix The constructed adjacency matrix is for The diagonal matrix of , and finally the ID embedding graph is obtained The function expression is: ， In the above formula, It means taking the average value, ~ They are No. 1~ The output of the layer is the ID embedding.

6. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 1, characterized in that: In step S2, the user-item graph of each modality is Use graph convolutional neural networks to extract high-order features to obtain modal embedding graphs of each modality include: S2.1, User-Item Graphs from Each Modality The high-level features are captured and projected into a unified embedding space: ， In the above formula, Display items No. The projection features of the modes, Indicates The modal transformation matrix of the modes is Display items No. The projection characteristics of each mode; S2.2, the user modal features are obtained by aggregating the projection features according to the following formula: ， In the above formula, For users No. The user modality features of the modality, For users In the user-item collaborative interaction matrix The set of neighbors in ; S2.3, construct the modal characteristic matrix of each mode according to the following formula: ， In the above formula, Indicates The modal characteristic matrix of the modes, ~ Respectively represent the 1st to User's The user modality features of the modality, is the number of users, ~ Respectively represent the 1st to Project No. The projection features of the modes, is the number of items; S2.4, extracting high-order features from the modal feature matrix of each modality using a graph convolutional neural network to obtain a modal embedding graph of each modality, wherein the graph convolutional neural network includes Layer, the modal feature matrix of each modality is the modal embedding of the input of the first layer , and the function expression of the graph propagation function used in each layer is: ， In the above formula, Indicates The output of the layer modality embedding of modalities, Indicates The output of the layer The modal embedding of each modality is performed The modal embedding combination of the modalities is obtained The modal embedding graph of the modal .

7. The multimodal recommendation method for enhancing user representation using graph convolutional neural networks according to claim 1, characterized in that: In step S3, the ID is embedded into the graph. And the modal embedding diagram of each mode Fusion gets the final embedding graph The function expression is: ， In the above formula, represents collaborative embedding, Indicates The modal embedding graph of the modes, represents the normalization function used to alleviate the difference in value scale between modal embedding maps, is the number of modes.

8. A multimodal recommendation system with enhanced user representation using a graph convolutional neural network, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multimodal recommendation method for enhancing user representation using a graph convolutional neural network as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the multimodal recommendation method for enhancing user representation by using a graph convolutional neural network as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the multimodal recommendation method for enhancing user representation by using a graph convolutional neural network as described in any one of claims 1 to 7 through a processor.

Images