Task-oriented multi-modal item recommendation method and apparatus, electronic device, and medium

Abstract

Embodiments of the present disclosure disclose a task-oriented multi-modal item recommendation method, device, electronic equipment and medium. A specific embodiment of the method comprises: obtaining an item modality dataset and a user-item interaction dataset; generating an initial item modality relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix; generating a training sample set, and performing the following training steps: inputting an initial interaction embedding representation set, a respective item modality embedding representation set, the initial item modality relationship graph matrix set and the user-item interaction matrix into an initial item recommendation model; determining an interaction prediction value corresponding to each training sample; generating an item recommendation loss value; in response to the item recommendation loss value being less than a preset loss threshold, determining the initial item recommendation model as an item recommendation model; and generating item recommendation information using the item recommendation model. The embodiment can improve the performance of the item recommendation model and recommend items of interest to the user to the user.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a task-oriented multimodal item recommendation method, apparatus, electronic device, and medium. Background Art

[0002] With the rapid development of multimodal learning technology, recommending multimodal items has become a key research topic in recommendation systems. Currently, the common approach for recommending multimodal items is as follows: First, in the modal representation extraction phase, pre-trained models (such as text and visual models) are used to extract item modal representations. Next, in the user interest modeling phase, the modal representations generated by the pre-trained models are integrated into the learning process of user and item interaction representations in the downstream recommendation task to generate the final user and item representations. Finally, item recommendations are generated based on the final user and item representations.

[0003] However, when using the above approach to recommend multimodal items, the following technical problems often arise:

[0004] Since the above-mentioned item recommendation method is divided into two stages: modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is usually not well adapted to the final interactive representation learning between users and items, which easily causes the model to fall into local optimality and reduces the accuracy of item representation. Furthermore, it is difficult to accurately determine the candidate items that the user is interested in, which makes it difficult to recommend the candidate items of interest to the user.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure provide task-oriented multimodal item recommendation methods, devices, electronic devices, and media to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a task-oriented multimodal item recommendation method, the method comprising: obtaining an item modal dataset and a user-item interaction dataset, wherein the item modal data in the above-mentioned item modal dataset corresponds to the items in the item set; generating an initial item modal relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix based on the above-mentioned item modal dataset and the above-mentioned user-item interaction dataset; generating a training sample set based on the above-mentioned user-item interaction dataset, and performing the following training steps based on the training sample set: inputting a pre-generated initial interaction embedding representation set, each item modal embedding representation set, an initial item modal relationship graph matrix set and a user-item interaction matrix into an initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second A graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, an item recommendation loss value is generated, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; using the above-mentioned item recommendation model, generating item recommendation information corresponding to the target user, and sending the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0009] In a second aspect, some embodiments of the present disclosure provide a task-oriented multimodal item recommendation device, the device comprising: an acquisition unit, configured to acquire an item modal dataset and a user-item interaction dataset, wherein the item modal data in the above-mentioned item modal dataset corresponds to the items in the item set; a construction unit, configured to generate an initial item modal relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix based on the above-mentioned item modal dataset and the above-mentioned user-item interaction dataset; a first generation and execution unit, configured to generate a training sample set based on the above-mentioned user-item interaction dataset, and to perform the following training steps based on the training sample set: inputting a pre-generated initial interaction embedding representation set, each item modal embedding representation set, an initial item modal relationship graph matrix set and a user-item interaction matrix into an initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network. network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; a second generation and sending unit is configured to use the above-mentioned item recommendation model to generate item recommendation information corresponding to the target user, and send the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0012] The aforementioned embodiments of the present disclosure have the following beneficial effects: The task-oriented multimodal item recommendation method of some embodiments of the present disclosure improves the accuracy of item recommendations and facilitates recommending items of interest to users. Specifically, the difficulty in recommending candidate items of interest to users arises from the fact that, because the aforementioned item recommendation method is divided into two stages, namely, modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is often not well adapted to the final learned representation of user-item interaction, which can easily cause the model to fall into a local optimum and reduce the accuracy of the item representation. Furthermore, it is difficult to accurately determine candidate items of interest to users, thereby making it difficult to recommend candidate items of interest to users. Based on this, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure first obtains an item modal dataset and a user-item interaction dataset. The item modal data in the item modal dataset corresponds to the items in the item set. This allows for obtaining item data representing different modal types of items and user-item interaction records, facilitating the subsequent learning of item modal representation constraints based on the user-item interaction records to generate a task-oriented item modal representation. Then, based on the aforementioned item modality dataset and the aforementioned user-item interaction dataset, we generate an initial item modality relationship graph matrix, a user-item interaction matrix, and an item second-order interaction matrix. This allows us to determine the relationships between items in different item modalities, the relationships under user-item interactions, and the interactions between users and items.Afterwards, based on the above-mentioned user-item interaction dataset, a training sample set is generated, and based on the training sample set, the following training steps are performed: the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item The method uses a pre-designed information constraint loss function and a recommendation loss function to generate an item recommendation loss value based on the initial item modal relationship graph matrix, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label. The pre-designed information constraint loss function and the recommendation loss function are configured to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items. In response to determining that the item recommendation loss value is less than a preset loss threshold, the trained initial item recommendation model is determined as the item recommendation model. This method generates an item recommendation model that includes high-order embedding representations of each user and high-order embedding representations of each item, facilitating the subsequent determination of items of interest to the user based on the high-order embedding representations of the user and items. Finally, the item recommendation model is used to generate item recommendation information for the target user and transmit the item recommendation information to the browsing terminal corresponding to the target user. This method allows the user to be recommended items of interest. Therefore, the task-oriented multimodal item recommendation methods of some embodiments of the present disclosure effectively match the two processes of modal representation learning and user-item interaction representation learning, unifying the learning of modal representation and user-item interaction representation from a graph perspective. Furthermore, through a constraint mechanism, the second-order interaction relationships between items extracted from user-item interaction relationships are used to guide the learning of modal representation. This can improve the precision and generalization ability of the item recommendation model, resulting in a higher accuracy of item representation. As a result, candidate items of user interest can be more accurately identified and recommended to the user. This, in turn, can improve the performance of the recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flowchart of some embodiments of the task-oriented multimodal item recommendation method according to the present disclosure;

[0015] Figure 2 is a schematic structural diagram of some embodiments of a task-oriented multimodal item recommendation device according to the present disclosure;

[0016] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 A process 100 of some embodiments of a task-oriented multimodal item recommendation method according to the present disclosure is shown. The task-oriented multimodal item recommendation method includes the following steps:

[0024] Step 101: Obtain an item modality dataset and a user-item interaction dataset.

[0025] In some embodiments, the execution entity (e.g., a computing device) of the task-oriented multimodal item recommendation method can obtain an item modality dataset and a user-item interaction dataset from a database via a wired or wireless connection. The item modality data in the item modality dataset can correspond one-to-one to each item in the item set. The item modality data in the item modality dataset can include an item identifier and a modality data set. The item identifier can be a unique identifier for the item. The modality data in the modality data set can include a modality identifier and item detail data. The modality identifier can be a unique identifier for the item modality. The item modality can be the type of data describing the item. For example, the item modality can be, but is not limited to, one of the following: image type, text type, or short video type. The item detail data can be descriptive data of the corresponding item in the corresponding modality. For example, the descriptive data can be an image of the item or text describing the item. The user-item interaction data in the user-item interaction dataset can represent interaction records between users and items. The user-item interaction data in the user-item interaction dataset can include, but is not limited to, a user identifier, an item identifier, and interaction time. The user identifier can be a unique identifier for the user. The aforementioned interaction time may be the start time of an interaction between the user and the object. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future developed wireless connection methods.

[0026] In practice, each item usually has descriptive data in multiple different modalities.

[0027] Step 102: Based on the item modality dataset and the user-item interaction dataset, generate an initial item modality relationship graph matrix set, a user-item interaction matrix, and an item second-order interaction relationship matrix.

[0028] In some embodiments, the execution entity may generate an initial item modal relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modal dataset and the user-item interaction dataset in various ways. The initial item modal relationship graph matrices in the initial item modal relationship graph matrix set may correspond one-to-one to item modalities. The initial item modal relationship graph matrices in the initial item modal relationship graph matrix set may be matrices to be updated, whose element values ​​represent the degree of similarity between modal data of items. The user-item interaction matrix may be a matrix whose element values ​​represent whether there is an interaction relationship between a user and an item. The inter-item second-order interaction relationship matrix may be a matrix whose element values ​​represent the similarity relationship between items.

[0029] In some optional implementations of some embodiments, the execution entity may generate an initial item modality relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modality dataset and the user-item interaction dataset through the following steps:

[0030] In the first step, a set of item modal data embedding information is generated based on the item modal dataset using a preset set of data embedding pretrained models. Each item modal data embedding information may include a modal data embedding information group. Each modal data embedding information may include a modality identifier and an item data embedding vector. The item data embedding vector may be an embedded representation of the item detail data of the corresponding item in the corresponding item modality. The data embedding pretrained models in the set of data embedding pretrained models may correspond one-to-one with the item modality. The data embedding pretrained models in the set of data embedding pretrained models may be pretrained models for embedding item detail data.

[0031] As an example, when the item modality is of text type, the corresponding data embedding pre-training model may be a language pre-training model. The language pre-training model may be, but is not limited to, one of the following: Sentence-BERT sentence embedding model, GPT model. When the item modality is of image type or short video type, the corresponding data embedding pre-training model may be a visual pre-training model. The visual pre-training model may be, but is not limited to, one of the following: ResNet residual network model, VIT (Vision Transformer) model. The above-mentioned execution subject may perform the following steps for each item modality data in the above-mentioned item modality dataset to generate the item modality data embedding information in the item modality data embedding information set:

[0032] Step 1: For each modal data included in the item modal data, the item details included in the modal data are input into a target data embedding pre-trained model to obtain an item data embedding vector, and the modal identifier corresponding to the modal data and the item data embedding vector are determined as modal data embedding information. The target data embedding pre-trained model may be a data embedding pre-trained model in the set of data embedding pre-trained models whose corresponding modal identifier is the same as the modal identifier included in the modal data.

[0033] Step 2: Determine the obtained modal data embedding information group and the item identifier included in the above-mentioned item modal data as item modal data embedding information.

[0034] The second step is to classify the individual item data embedding vectors included in the item modal data embedding information set to obtain a set of homomodal item data embedding vector groups. The homomodal item data embedding vector groups in the homomodal item data embedding vector group set may be individual item data embedding vectors corresponding to the same item modality. The individual item data embedding vectors included in the item modal data embedding information set may be classified based on the modality identifier corresponding to each item data embedding vector to obtain a set of homomodal item data embedding vector groups.

[0035] The third step is to construct an initial item modal relationship graph matrix set based on the above-mentioned set of embedding vectors for the same-modal item data. The initial item modal relationship graph matrix set can be constructed based on the above-mentioned set of embedding vectors for the same-modal item data in various ways.

[0036] In some optional implementations of some embodiments, the execution entity may perform the following steps for each homomodal item data embedding vector group in the homomodal item data embedding vector group set to construct an initial item modal relationship graph matrix in the initial item modal relationship graph matrix set:

[0037] Step 1: Determine the item modal data similarity set corresponding to the same-modal item data embedding vector set. Each item modal data similarity can represent the degree of similarity between two different items under the corresponding item modality. The item modal data similarity between each pair of same-modal item data embedding vectors in the same-modal item data embedding vector set can be determined using cosine similarity to obtain the item modal data similarity set.

[0038] Step 2: Construct an item modal relationship graph using the items in the item set as nodes and the modal data similarities of the items in the item modal data similarity set as the weights of the edges between nodes. The item modal relationship graph can be a fully connected graph that represents the modal similarity between items. The item modal relationship graph often contains a large number of unimportant edges.

[0039] Step three, perform graph sparsification processing on the above-mentioned item modal relationship graph to obtain a sparse item modal relationship graph. The above-mentioned sparse item modal relationship graph may be a sparse graph with fewer edges than the item modal relationship graph. First, sort the edge weights of the above-mentioned item modal relationship graph in descending order to obtain a sorted edge weight sequence. Then, the sorted edge weights that meet the preset sequence number conditions in the above-mentioned sorted edge weight sequence are determined as reference weights. Finally, according to the graph sparsification technology, the edges in the edge set corresponding to the above-mentioned item modal relationship graph whose corresponding weights are less than the above-mentioned reference weights can be deleted to obtain the deleted item modal relationship graph as the sparse item modal relationship graph.

[0040] Step 4: Based on the sparse item modal relationship graph, construct an initial item modal relationship graph matrix. First, construct an initial adjacency matrix with the items in the item set as rows and columns and zero as the element. Then, based on the weights of the corresponding edges between row and column items in the sparse item modal relationship graph, update the elements at the corresponding positions in the initial adjacency matrix with the weight values ​​to obtain the initial item modal relationship graph matrix.

[0041] The fourth step is to determine the user set, interaction item set, and user-item pair set corresponding to the user-item interaction dataset. First, all unique users in the user-item interaction dataset are identified as a user set. Then, all unique items in the user-item interaction dataset are identified as an interaction item set. Finally, for each user-item interaction data set in the user-item interaction dataset, the user and interaction item included in the user-item interaction data are identified as a user-item pair. It should be noted that the item set can be larger than the interaction item set.

[0042] Step 5: Construct a user-item interaction matrix based on the user-item pair set, with the users in the user set as rows and the interactive items in the interactive item set as columns. The elements in the user-item interaction matrix represent whether there is an interaction between the corresponding user and the interactive item. For example, a 1 in the user-item interaction matrix indicates an interaction between the corresponding user and the interactive item, while a 0 indicates no interaction between the corresponding user and the interactive item.

[0043] The sixth step is to extract the second-order relationship between items from the above user-item interaction matrix to obtain the second-order interaction relationship matrix between items. The second-order relationship extraction step can be performed using the following formula:

[0044] S=R T ×R.

[0045] Where R represents the user-item interaction matrix. T represents the transposed matrix of the user-item interaction matrix. S represents the second-order interaction matrix between items.

[0046] In practice, the above-mentioned execution entity extracts the potential interaction relationship between items from the interaction behavior patterns of users and items, which can be used to subsequently guide the update of the item modal relationship graph matrix, thereby facilitating the full integration of the two processes of user interest modeling and item modal relationship extraction.

[0047] Step 103: Generate a training sample set based on the user-item interaction dataset, and perform the following training steps based on the training sample set:

[0048] In step 1031, the pre-generated initial interaction embedding representation set, each item modality embedding representation set, the initial item modality relationship graph matrix set, and the user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set.

[0049] In some embodiments, the execution entity may input a pre-generated set of initial interaction embedding representations, sets of embedding representations for each item modality, an initial item modality relationship graph matrix, and a user-item interaction matrix into the initial item recommendation model to obtain a set of high-order user embedding vectors and a set of high-order item embedding vectors. The initial interaction embedding representation set may include an initial user embedding vector set and an initial item embedding vector set. The initial user embedding vectors in the initial user embedding vector set may correspond one-to-one with users in the user set. The initial user embedding vectors in the initial user embedding vector set may be initialized embedding representations of the corresponding users. The initial item embedding vectors in the initial item embedding vector set may be initialized embedding representations of the corresponding items. For example, the initial user embedding vectors and the initial item embedding vectors may both be 32-dimensional vectors, with each dimension initially set to 0. The item modality embedding representations in each item modality embedding representation set may correspond one-to-one with items in the item set. The item modality embedding representations in the item modality embedding representation set may correspond one-to-one with item modalities. The item modality embedding representations in each item modality embedding representation set may be initialized sets of embedding representations corresponding to different modal data of the corresponding items. For example, the modal embedding representation of an item can be a 32-dimensional vector, and each dimension can be initially 0. The initial item recommendation model can include a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule, and an item embedding representation fusion layer. The above-mentioned first graph neural network submodule can be used to output the initial high-order interaction representation of the user and the item. The above-mentioned second graph neural network submodule can be used to output the high-order modal representation of the item. The graph neural network used in the above-mentioned first graph neural network submodule has a similar structure to the graph neural network used in the above-mentioned second graph neural network submodule. The above-mentioned graph neural network can be a graph convolutional network. The above-mentioned multi-head attention submodule can be used to fuse item representations under different modalities. The above-mentioned item embedding representation fusion layer can be used to fuse the interaction representation and modal representation of the item.

[0050] In some optional implementations of some embodiments, the execution entity may input a pre-generated initial interaction embedding representation set, each item modality embedding representation set, an initial item modality relationship graph matrix set, and a user-item interaction matrix into an initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set through the following steps:

[0051] In the first step, the pre-generated initial interaction embedding representation set and the user-item interaction matrix are input into the first graph neural network submodule included in the initial item recommendation model to obtain a high-order interaction embedding representation set. The above-mentioned high-order interaction embedding representation set may include a high-order user embedding vector set and an initial high-order item embedding vector set. The high-order user embedding vectors in the above-mentioned high-order user embedding vector set may correspond one-to-one to users in the user set. The initial high-order item embedding vectors in the above-mentioned initial high-order item embedding vector set may correspond one-to-one to items in the above-mentioned item set. Each high-order user embedding vector may be a high-order interaction representation of the corresponding user. Each initial high-order item embedding vector may be a high-order interaction representation to be updated for the corresponding item. It should be noted that the high-order interaction representation may be a high-order embedding vector generated by the target node continuously aggregating embedding vectors from neighboring nodes and obtaining information from multi-hop (multi-order) neighboring nodes in the process of using graph neural network technology to learn the embedding vectors of nodes (users or items) in the graph.

[0052] As an example, the processing of the first graph neural network submodule can be expressed by the following formula:

[0053]

[0054] Where r represents the interaction between users and items. l represents the number of layers in the graph neural network. l+1 represents the l+1th layer of the graph neural network. The value of l can range from 0 to an upper limit of L. L represents the total number of layers in the graph neural network. GNN(·) represents the graph neural network. and They represent the sets of interactive embedding representations of the l+1th layer and the lth layer respectively. The initial input of the graph neural network is the initial interaction embedding representation set E r . L r Represents the normalized adjacency matrix R. And D represents the degree matrix. Finally, the set of high-order interaction embedding representations of users and items can be determined by the following formula:

[0055]

[0056] Among them, the high-order interaction embedding representation set H r Includes a collection of high-level user embedding vectors and the initial high-order item embedding vector set express u represents a user. |U| represents the number of users. represents the high-order user embedding vector corresponding to the |U|th user. i represents the item. |I| represents the number of items. express Represents the initial high-order item embedding vector corresponding to the |I|th item.

[0057] In practice, this solution can not only fully capture the potential relationship between users and items by leveraging the information propagation mechanism of graph neural networks, but can also further extract second-order item-item interaction relationships from the user-item interaction matrix R to guide the learning of item modal representations in the modal relationship graph. Therefore, to learn item modal representations from the modal relationship graph, this disclosure decides to adopt the following steps:

[0058] In the second step, the pre-generated set of item modal embedding representations and the initial item modal relationship graph matrix set are input into the second graph neural network sub-module included in the initial item recommendation model to obtain a set of high-order item modal embedding vector groups. Each high-order item modal embedding vector group in the above-mentioned high-order item modal embedding vector group set corresponds one-to-one to an item in the above-mentioned item set. Each high-order item modal embedding vector in the high-order item modal embedding vector group corresponds one-to-one to an item modality. Each high-order item modal embedding vector group in the above-mentioned high-order item modal embedding vector group set can be a high-order modal representation under different item modalities of the corresponding item.

[0059] As an example, for any item modality, the processing of the second graph neural network submodule can be expressed by the following formula:

[0060]

[0061] Where m represents the serial number of the item mode. and Represent the modal representation sets of items at the l+1th layer and the lth layer respectively. The initial input of the graph neural network is the item modality embedding representation set E m . A m Represents the initial item modality relationship graph matrix corresponding to item modality m. Indicates A m Normalized adjacency matrix. And B represents the degree matrix. Finally, the high-order modal representation of each item can be determined by the following formula:

[0062]

[0063] Among them, H m When the item modality corresponding to each item is m item modality, the high-order modality representation of each item (high-order item modality embedding vector) should be noted that A m For dynamic learnable matrix parameters, in each iteration of the model, the adjacency matrix A mThe updating process is similar to the construction process of the item modal relationship diagram. The difference is that the present disclosure uses the high-order item modal representation H updated in each iteration process. m To construct the adjacency matrix A of the item modal relationship graph m This construction process is the same as that of the initial item modality relationship graph matrix. Furthermore, the second graph neural network submodule can also use the high-order item modality embedding vectors under different item modalities corresponding to each item as a high-order item modality embedding vector group to obtain a set of high-order item modality embedding vector groups corresponding to each item.

[0064] The third step is to input the above-mentioned high-order item modal embedding vector set into the multi-head attention submodule included in the initial item recommendation model to obtain the item multimodal embedding vector set. Among them, each item multimodal embedding vector set in the above-mentioned item multimodal embedding vector set can correspond one-to-one to the high-order item modal embedding vector set in the above-mentioned high-order item modal embedding vector set. Each item multimodal embedding vector in the above-mentioned item multimodal embedding vector set can be a vector after fusing the different modal representations of the corresponding item. According to the multi-head attention mechanism, the item multimodal embedding vector in the above-mentioned item multimodal embedding vector set can be generated by the following formula:

[0065]

[0066] Where M represents the set of item modalities. |M| represents the number of item modalities. m′ represents an item modality in the set M that is distinct from the mth item modality, i.e., m ≠ m′. P represents the number of attention heads. P represents the sequence number of the attention head. d represents the dimension of the item modality representation. Indicates the multimodal embedding vector of the item when the item detail data of item i is of the mth item modality. Indicates the multimodal embedding vector of the item when the item detail data of item i is of the m′th item modality. represents the multimodal embedding vector of the item after the multi-head attention mechanism. ρ(·) represents the softmax activation function. and They represent the multi-head query and key conversion matrices in the multi-head attention mechanism, and

[0067] Step 4: For each item multimodal embedding vector group in the above item multimodal embedding vector group set, the above item multimodal embedding vector group and the corresponding initial high-order item embedding vector are input into the item embedding representation fusion layer included in the initial item recommendation model to obtain a high-order item embedding vector. The high-order item embedding vector is the embedding representation of the item after the high-order interaction representation and modal representation of the corresponding item are fused. The high-order item embedding vector corresponding to each item can be generated through the item embedding representation fusion layer according to the following formula:

[0068]

[0069] Among them, h i Represents the high-order item embedding vector corresponding to item i. Represents the initial high-order item embedding vector corresponding to item i.

[0070] It should be noted that the training samples in the above training sample set consist of data from one user and one item. The training sample set includes a positive training sample set and a negative training sample set. The positive training samples in the positive training sample set can represent actual interaction records between the user and the item. The negative training samples in the negative training sample set can represent the absence of actual interaction records between the user and the item. A training sample set can be generated based on the user-item interaction dataset by performing the following steps:

[0071] In the first step, for each user-item interaction data in the user-item interaction dataset, the user identifier and item identifier included in the user-item interaction data are determined as positive training samples.

[0072] The second step is to perform negative sampling on each of the obtained positive training samples to obtain a negative training sample set. First, for each positive training sample, items that have not been interacted with by the user included in the positive training sample can be sampled. Each sampled item and the user included in the positive training sample can be combined to form a negative training sample, thereby obtaining a negative training sample group. Then, each negative training sample in each of the obtained negative training sample groups is determined as a negative training sample set.

[0073] Step 1032: Determine an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set based on the high-order user embedding vector set and the high-order item embedding vector set.

[0074] In some embodiments, the execution entity may determine an interaction prediction value corresponding to each of at least one training sample in the training sample set based on the high-order user embedding vector set and the high-order item embedding vector set. The interaction prediction value may represent a probability of a possible interaction between the user and the item. The interaction prediction value corresponding to each of at least one training sample in the training sample set may be determined according to the following interaction prediction formula:

[0075]

[0076] in, represents the interaction prediction value between user u and item i. δ(·) represents the sigmoid activation function. Represents the high-order user embedding vector corresponding to user u.

[0077] Continuing, considering that in a recommendation system that integrates multimodal item information, users typically comprehensively evaluate whether to interact with an item based on its modal information, and further, the modal connection relationship between items is also reflected in the second-order item-item interaction relationship. Therefore, in order to obtain the potential connection between the modal connection relationship between items and the second-order item-item interaction relationship, and to inject the second-order item-item interaction information into the modal relationship graph by narrowing the information gap between the two matrices to constrain the learning of the item modal representation, the present disclosure has decided to introduce a pre-designed information constraint loss function into the following model loss calculation step 1033.

[0078] Step 1033: Generate an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in at least one training sample, and the corresponding sample label through the pre-designed information constraint loss function and recommendation loss function.

[0079] In some embodiments, the execution entity may generate an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label using a pre-designed information constraint loss function and recommendation loss function. The information constraint loss function may be used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items. The recommendation loss function may be used to minimize the difference between the probability distribution of the interaction prediction value and the probability distribution of the true label. The sample label may be 1 or 0. When a training sample is a positive training sample, the corresponding sample label may be 1. When a training sample is a negative training sample, the corresponding sample label may be 0. The item recommendation loss value may represent the degree of difference between the predicted user-item interaction result and the actual user-item interaction result.

[0080] In some optional implementations of some embodiments, the execution entity may generate an item recommendation loss value based on the initial item modality relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label by using a pre-designed information constraint loss function and a recommendation loss function through the following steps:

[0081] In the first step, the information constraint loss function is used to determine the first loss value between the initial item modal relationship graph matrix set and the above-mentioned second-order interaction relationship matrix between items. The first loss value represents the information difference between the initial item modal relationship graph matrix and the second-order interaction relationship matrix between items under each item modality. The information constraint loss function can be:

[0082]

[0083] in, Represents the information constraint loss value. represents the normalized adjacency matrix of S. ||·|| F represents the Frobenius norm.

[0084] As an example, the execution entity may first determine, for each initial item modal relationship graph matrix in the initial item modal relationship graph matrix set, an information constraint loss value between the initial item modal relationship graph matrix and the inter-item second-order interaction relationship matrix based on the information constraint loss function. The sum of the obtained information constraint loss values ​​may then be determined as the first loss value.

[0085] In addition, the present disclosure considers that KL divergence can also be used to measure the degree of difference between two probability distributions, and therefore designs another constraint mechanism, namely, by minimizing and The KL divergence between them is used to constrain the adjacency matrix of the modal relationship graph To guide the learning of item modal representations. The constraint mechanism based on KL divergence can be represented by the following information constraint loss function.

[0086] Optionally, the above information constraint loss function can also be:

[0087]

[0088] Where N is the number of items. n is the item number. KL(·||·) represents the KL divergence of two discrete distributions. Represents the adjacency matrix The row vector corresponding to the nth item in . Represents the adjacency matrix The row vector corresponding to the nth item in .

[0089] The information constraint loss functions of the above two methods and their related contents, as the inventive points of the present disclosure, can evaluate the modal relationship between items and the second-order item-item interaction relationship in the modal relationship graph, and by minimizing the information constraint loss value, utilize the interactive representation learning of users and items in the recommendation task to constrain the modal representation learning process in the modal relationship graph, so that the generated item modal representation contains the interactive information between users and items, and finally generate a task-oriented item modal representation.

[0090] In the second step, a second loss value between the interaction prediction value corresponding to each training sample in the at least one training sample and the corresponding sample label is determined by recommending a loss function.

[0091] Among them, the above recommendation loss function can be:

[0092]

[0093] in, represents the second loss value. X represents the training sample set. y u，i Represents the sample label.

[0094] The third step is to generate a recommended loss value for the item based on the first loss value and the second loss value. The recommended loss value for the item can be generated based on the first loss value and the second loss value using the following formula:

[0095]

[0096] in, Indicates the item recommendation loss value. represents the regularization term. α represents the weight coefficient of the information constraint loss. β represents the weight coefficient of the regularization term.

[0097] Step 1034 : In response to determining that the item recommendation loss value is less than the preset loss threshold, the trained initial item recommendation model is determined as the item recommendation model.

[0098] In some embodiments, the execution entity may determine the trained initial item recommendation model as the item recommendation model in response to determining that the item recommendation loss value is less than a preset loss threshold. The preset loss threshold may be a preset upper limit of the item recommendation loss value.

[0099] Optionally, in response to determining that the item recommendation loss value is greater than or equal to the preset loss threshold, the execution entity may further adjust trainable network parameters in the initial item recommendation model, use unused training samples to form a training sample set, and re-perform the training step using the adjusted initial item recommendation model. The trainable network parameters may include an initial interaction embedding representation set and an initial item-modality relationship graph matrix set.

[0100] As an example, the above-mentioned execution entity can use the Tensorflow framework to build an item recommendation model. During the item recommendation model training phase, the above-mentioned execution entity optimizes the trainable parameters of the model through the Adam optimizer, and sets the initial learning rate to 0.001. The amount of training data for each batch is set to 128, the interaction representation dimension between users and items is set to 32, and the modal representation dimension of items is also set to 32. To prevent overfitting, the present disclosure also uses dropout technology to randomly remove nodes in the graph structure with a probability of 0.3. The number of layers of the graph neural network used in the item recommendation model is set to 3. When calculating the item recommendation loss value, the weight coefficient α is set to 0.02 and β is set to 0.0001.

[0101] Step 104 : Generate item recommendation information corresponding to the target user using the item recommendation model, and send the item recommendation information to the browsing terminal corresponding to the target user.

[0102] In some embodiments, the execution entity may utilize the item recommendation model in various ways to generate item recommendation information corresponding to the target user, and send the item recommendation information to the browsing terminal corresponding to the target user. The target user may be a user who has logged into the item display platform on which the item recommendation model is deployed. The item display platform may be a platform for displaying various items. For example, the item display platform may be a shopping website or an online e-book reading platform. The item recommendation information may be information about various items to be recommended to the user for interactive display. The browsing terminal may be a terminal with a display screen used by the target user when logging into the item display platform.

[0103] In some optional implementations of some embodiments, the execution entity may generate item recommendation information corresponding to the target user using the item recommendation model through the following steps:

[0104] The first step is to determine a set of non-interacted item identifiers corresponding to the target user based on the user-item interaction dataset and the item set. The non-interacted item identifiers in the non-interacted item identifier set may represent items that have not been interacted with by the target user.

[0105] The second step is to determine a target high-order user embedding vector corresponding to the target user based on the high-order user embedding vector set. The high-order user embedding vector corresponding to the target user can be selected from the high-order user embedding vector set as the target high-order user embedding vector.

[0106] Step 3: For each uninteracted item identifier in the uninteracted item identifier set, perform the following steps:

[0107] The first sub-step is to determine a target high-order item embedding vector corresponding to the non-interacted item identifier based on the set of high-order item embedding vectors. The high-order item embedding vector corresponding to the non-interacted item identifier can be selected from the set of high-order item embedding vectors as the target high-order item embedding vector.

[0108] In the second sub-step, the item recommendation model is used to determine an interaction prediction rate based on the target high-order user embedding vector corresponding to the target user and the target high-order item embedding vector corresponding to the non-interacted item identifier. The interaction prediction rate may be the probability of a possible interaction between the user and the item. The interaction prediction rate can be determined using the interaction prediction formula in the item recommendation model based on the target high-order user embedding vector corresponding to the target user and the target high-order item embedding vector corresponding to the non-interacted item identifier.

[0109] In the third sub-step, the non-interacted item identifier and the interaction prediction rate are determined as candidate item information.

[0110] In the fourth step, the determined candidate item information is sorted in descending order to obtain a candidate item information sequence. The candidate item information sequence can be obtained by sorting the determined candidate item information in descending order using a preset sorting algorithm. For example, the sorting algorithm can be, but is not limited to, one of: bubble sort and insertion sort.

[0111] The fifth step is to generate item recommendation information corresponding to the target user based on the candidate item information sequence. First, candidate item information with corresponding serial numbers less than or equal to a preset serial number threshold can be selected from the candidate item information sequence as target candidate item information to obtain a target candidate item information group. The preset serial number threshold can be a pre-set upper limit of the serial number. Then, for each target candidate item information in the target candidate item information group, the target item information corresponding to the target candidate item information is determined as the item information to be recommended. The target item information can be the item details information of the item corresponding to the uninteracted item identifier in the target candidate item information. Finally, each item information to be recommended is determined as the item recommendation information.

[0112] The aforementioned embodiments of the present disclosure have the following beneficial effects: The task-oriented multimodal item recommendation method of some embodiments of the present disclosure improves the accuracy of item recommendations and facilitates recommending items of interest to users. Specifically, the difficulty in recommending candidate items of interest to users arises from the fact that, because the aforementioned item recommendation method is divided into two stages, namely, modal representation extraction and user interest modeling, the item modal representation output by the pre-trained model is often not well adapted to the final learned representation of user-item interaction, which can easily cause the model to fall into a local optimum and reduce the accuracy of the item representation. Furthermore, it is difficult to accurately determine candidate items of interest to users, thereby making it difficult to recommend candidate items of interest to users. Based on this, the task-oriented multimodal item recommendation method of some embodiments of the present disclosure first obtains an item modal dataset and a user-item interaction dataset. The item modal data in the item modal dataset corresponds to the items in the item set. This allows for obtaining item data representing different modal types of items and user-item interaction records, facilitating the subsequent learning of item modal representation constraints based on the user-item interaction records to generate a task-oriented item modal representation. Then, based on the aforementioned item modality dataset and the aforementioned user-item interaction dataset, we generate an initial item modality relationship graph matrix, a user-item interaction matrix, and an item second-order interaction matrix. This allows us to determine the relationships between items in different item modalities, the relationships under user-item interactions, and the interactions between users and items.Afterwards, based on the above-mentioned user-item interaction dataset, a training sample set is generated, and based on the training sample set, the following training steps are performed: the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item The method uses a pre-designed information constraint loss function and a recommendation loss function to generate an item recommendation loss value based on the initial item modal relationship graph matrix, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label. The pre-designed information constraint loss function and the recommendation loss function are configured to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items. In response to determining that the item recommendation loss value is less than a preset loss threshold, the trained initial item recommendation model is determined as the item recommendation model. This method generates an item recommendation model that includes high-order embedding representations of each user and high-order embedding representations of each item, facilitating the subsequent determination of items of interest to the user based on the high-order embedding representations of the user and items. Finally, the item recommendation model is used to generate item recommendation information for the target user and transmit the item recommendation information to the browsing terminal corresponding to the target user. This method allows the user to be recommended items of interest. Therefore, the task-oriented multimodal item recommendation methods of some embodiments of the present disclosure effectively match the two processes of modal representation learning and user-item interaction representation learning, unifying the learning of modal representation and user-item interaction representation from a graph perspective. Furthermore, through a constraint mechanism, the second-order interaction relationships between items extracted from user-item interaction relationships are used to guide the learning of modal representation. This can improve the precision and generalization ability of the item recommendation model, resulting in a higher accuracy of item representation. As a result, candidate items of user interest can be more accurately identified and recommended to the user. This, in turn, can improve the performance of the recommendation system.

[0113] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a task-oriented multimodal item recommendation device. These device embodiments are similar to Figure 1Corresponding to the method embodiments shown, the task-oriented multimodal item recommendation apparatus 200 can be specifically applied to various electronic devices.

[0114] like Figure 2 As shown, the task-oriented multimodal item recommendation device 200 of some embodiments includes: an acquisition unit 201, a construction unit 202, a first generation and execution unit 203 and a second generation and sending unit 204. Among them, the acquisition unit 201 is configured to acquire an item modal data set and a user-item interaction data set, wherein the item modal data in the above-mentioned item modal data set corresponds to the items in the item set; the construction unit 202 is configured to generate an initial item modal relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix based on the above-mentioned item modal data set and the above-mentioned user-item interaction data set; the first generation and execution unit 203 is configured to generate a training sample set based on the above-mentioned user-item interaction data set, and perform the following training steps based on the training sample set: input the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule block, a multi-head attention sub-module and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; the second generating and sending unit 204 is configured to use the above-mentioned item recommendation model to generate item recommendation information corresponding to the target user, and send the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0115] It is understandable that the various units recorded in the task-oriented multimodal item recommendation device 200 and the reference Figure 1Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the task-oriented multimodal item recommendation device 200 and the units included therein, and will not be described in detail here.

[0116] Further references Figure 3 , which shows a structural diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0117] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0118] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0119] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0120] It should be noted that in some embodiments of the present disclosure, the computer-readable medium mentioned above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0121] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0122] The above-mentioned computer-readable medium may be included in the above-mentioned device; or it may exist independently and not be assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains an item modal data set and a user-item interaction data set, wherein the item modal data in the above-mentioned item modal data set corresponds to the items in the item set; based on the above-mentioned item modal data set and the above-mentioned user-item interaction data set, generates an initial item modal relationship graph matrix set, a user-item interaction matrix and an inter-item second-order interaction relationship matrix; based on the above-mentioned user-item interaction data set, generates a training sample set, and based on the training sample set, performs the following training steps: inputs the pre-generated initial interaction embedding representation set, each item modal embedding representation set, the initial item modal relationship graph matrix set and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, wherein the initial interaction embedding representation set includes an initial user embedding vector set and an initial item embedding vector set, and the initial item recommendation model includes a first graph neural network sub-set. module, a second graph neural network submodule, a multi-head attention submodule and an item embedding representation fusion layer; based on the above-mentioned high-order user embedding vector set and the above-mentioned high-order item embedding vector set, determining the interaction prediction value corresponding to each training sample in the at least one training sample included in the training sample set; through a pre-designed information constraint loss function and a recommendation loss function, based on the initial item modal relationship graph matrix set, the above-mentioned second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the above-mentioned at least one training sample and the corresponding sample label, generating an item recommendation loss value, wherein the above-mentioned information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the above-mentioned second-order interaction relationship matrix between items; in response to determining that the above-mentioned item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; using the above-mentioned item recommendation model, generating item recommendation information corresponding to the target user, and sending the above-mentioned item recommendation information to the browsing terminal corresponding to the above-mentioned target user.

[0123] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units described in some embodiments of the present disclosure may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor comprising: an acquisition unit, a construction unit, a first generation and execution unit, and a second generation and sending unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the acquisition unit may also be described as a "unit for acquiring an item modal dataset and a user-item interaction dataset."

[0126] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0127] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A task-oriented multimodal item recommendation method, comprising: Obtaining an item modality dataset and a user-item interaction dataset, wherein the item modality data in the item modality dataset corresponds to items in the item set; Based on the item modality dataset and the user-item interaction dataset, generating an initial item modality relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix; Based on the user-item interaction dataset, a training sample set is generated, and based on the training sample set, the following training steps are performed: Input the pre-generated initial interaction embedding representation set, each item modality embedding representation set, the initial item modality relationship graph matrix set, and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set. The initial interaction embedding representation set includes the initial user embedding vector set and the initial item embedding vector set. The initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule, and an item embedding representation fusion layer. Determining, based on the high-order user embedding vector set and the high-order item embedding vector set, an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; Generate an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label using a pre-designed information constraint loss function and recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items; In response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; Utilizing the item recommendation model, item recommendation information corresponding to a target user is generated, and the item recommendation information is sent to a browsing terminal corresponding to the target user.

2. The method according to claim 1, wherein The method further comprises: In response to determining that the item recommendation loss value is greater than or equal to the preset loss threshold, adjusting the trainable network parameters in the initial item recommendation model, and using unused training samples to form a training sample set, using the adjusted initial item recommendation model, and performing the training step again, wherein the trainable network parameters include an initial interaction embedding representation set and an initial item modal relationship graph matrix set.

3. The method according to claim 1, wherein The generating of an initial item modality relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modality dataset and the user-item interaction dataset includes: Generate an item modal data embedding information set based on the item modal data set using a preset data embedding pre-training model set, wherein each item modal data embedding information includes a modal data embedding information group, and each modal data embedding information includes a modality identifier and an item data embedding vector; Classifying each item data embedding vector included in the item modal data embedding information set to obtain a set of same-modal item data embedding vectors; Based on the same-modality item data embedding vector group set, constructing an initial item modality relationship graph matrix set; Determining a user set, an interaction item set, and a user-item pair set corresponding to the user-item interaction dataset; Constructing a user-item interaction matrix based on the user set and the interactive items in the interactive item set as columns, and based on the user-item pair set; The user-item interaction matrix is ​​subjected to an item second-order relationship extraction process to obtain an inter-item second-order interaction relationship matrix.

4. The method according to claim 3, wherein: The step of constructing an initial item modality relationship graph matrix set based on the same-modality item data embedding vector set includes: For each homomodal item data embedding vector group in the homomodal item data embedding vector group set, perform the following steps: Determining a set of item modal data similarities corresponding to the same-modal item data embedding vector group, wherein each item modal data similarity represents the degree of similarity between the corresponding modalities of two different items; Constructing an item modality relationship graph using items in the item set as nodes and item modality data similarities in the item modality data similarity set as weights of edges between nodes; Performing graph sparsification processing on the item modal relationship graph to obtain a sparse item modal relationship graph; Based on the sparse item modal relationship graph, an initial item modal relationship graph matrix is ​​constructed.

5. The method according to any one of claims 1 to 4, wherein: The pre-generated initial interaction embedding representation set, each item modality embedding representation set, initial item modality relationship graph matrix set, and user-item interaction matrix are input into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set, including: Inputting the pre-generated initial interaction embedding representation set and the user-item interaction matrix into the first graph neural network sub-module included in the initial item recommendation model to obtain a high-order interaction embedding representation set, wherein the high-order interaction embedding representation set includes a high-order user embedding vector set and an initial high-order item embedding vector set, and the initial high-order item embedding vectors in the initial high-order item embedding vector set correspond to the items in the item set; Inputting the pre-generated set of item modal embedding representations and the initial item modal relationship graph matrix set into the second graph neural network sub-module included in the initial item recommendation model to obtain a set of high-order item modal embedding vector groups, wherein each high-order item modal embedding vector group in the set of high-order item modal embedding vector groups corresponds to an item in the item set; Inputting the high-order item modal embedding vector set into a multi-head attention submodule included in the initial item recommendation model to obtain an item multimodal embedding vector set, wherein each item multimodal embedding vector set in the item multimodal embedding vector set corresponds to a high-order item modal embedding vector set in the high-order item modal embedding vector set; For each item multimodal embedding vector group in the item multimodal embedding vector group set, the item multimodal embedding vector group and the corresponding initial high-order item embedding vector are input into the item embedding representation fusion layer included in the initial item recommendation model to obtain a high-order item embedding vector.

6. The method according to claim 1, wherein Generating item recommendation information corresponding to a target user by using the item recommendation model includes: Determining a non-interacted item identification set corresponding to the target user based on the user-item interaction dataset and the item set; Determining a target high-order user embedding vector corresponding to the target user based on the high-order user embedding vector set; For each non-interacted item identifier in the non-interacted item identifier set, perform the following steps: Determining a target high-order item embedding vector corresponding to the non-interacted item identifier based on the high-order item embedding vector set; Determining an interaction prediction rate based on a target high-order user embedding vector corresponding to the target user and a target high-order item embedding vector corresponding to the non-interacted item identifier by the item recommendation model; determining the non-interacted item identifier and the interaction prediction rate as candidate item information; Arrange the determined candidate item information in descending order to obtain a candidate item information sequence; Based on the candidate item information sequence, item recommendation information corresponding to the target user is generated.

7. A task-oriented multimodal item recommendation device, comprising: an acquisition unit configured to acquire an item modality dataset and a user-item interaction dataset, wherein the item modality data in the item modality dataset corresponds to items in the item set; A construction unit is configured to generate an initial item modality relationship graph matrix set, a user-item interaction matrix, and an inter-item second-order interaction relationship matrix based on the item modality dataset and the user-item interaction dataset; The first generating and executing unit is configured to generate a training sample set based on the user-item interaction dataset, and execute the following training steps based on the training sample set: Input the pre-generated initial interaction embedding representation set, each item modality embedding representation set, the initial item modality relationship graph matrix set, and the user-item interaction matrix into the initial item recommendation model to obtain a high-order user embedding vector set and a high-order item embedding vector set. The initial interaction embedding representation set includes the initial user embedding vector set and the initial item embedding vector set. The initial item recommendation model includes a first graph neural network submodule, a second graph neural network submodule, a multi-head attention submodule, and an item embedding representation fusion layer. Determining, based on the high-order user embedding vector set and the high-order item embedding vector set, an interaction prediction value corresponding to each training sample in at least one training sample included in the training sample set; Generate an item recommendation loss value based on the initial item modal relationship graph matrix set, the second-order interaction relationship matrix between items, the interaction prediction value corresponding to each training sample in the at least one training sample, and the corresponding sample label using a pre-designed information constraint loss function and recommendation loss function, wherein the information constraint loss function is used to minimize the information difference between each initial item modal relationship graph matrix and the second-order interaction relationship matrix between items; In response to determining that the item recommendation loss value is less than a preset loss threshold, determining the trained initial item recommendation model as the item recommendation model; The second generating and sending unit is configured to generate item recommendation information corresponding to the target user by using the item recommendation model, and send the item recommendation information to the browsing terminal corresponding to the target user.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images